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. A study entitled the Alternative Training Experiment 

(ATE) evaluated the effectiveness of a method for training census 
enumerators that involved new processes for developing instructional 
materials (called job performance a'us) and new mot hods of 
instruction. During the study three pairs of census decentralized 
offices were matched on variables related to the difficulty of 
enumeration. Half the trainees from each office were given the 
experimental or Job-Performance-Aided (JPA) Training, while the 
remaining trainees were trained via conventional training methods. 
Based on enumerator reactions collected immediately after the 
training and again after two to three days of job experience, 
researchers found a slight but consistent preference for JPA 
training. A second study compared the job performance of the two 
groups of participants on a variety of performance measures. After 
analyzing data from end-of-training knowledge tests, self-report 
measures, and quality control records obtained as part of normal 
field operations, researchers decided that no conclusions about the 
effects of training on job performance were possible. However, they 
did make several recommendations concerning training development and 
research, including designing training packages with some flex time, 
and making the design of training and field manuals an integrated 
effort. (MN) 



*********************************************************************** 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 
*********************************************************************** 



9 

ERIC 



CO 

CD 
r-* 
sO 

CM 



An Evaluation of the Use of Job Aids in 1980 Decennial Census Training 



OS 



9 

ERIC 



Prepared by: 



William P. Mockovak, Ph.D. 
Research Psychologist 
Center for Social Science Research 
Bureau of the Census 
Washington, D.C. 



o 



US DEPARTMENT OF EDUCATION 

NATIONAL INSTITUTE OF EDUCATION 

EDUCATIONAL RESOURCES INFORMATION 

/f CENTER IERIC) 

This document has oeen reproduced as 
receded from the person or organization 
originating it 

Minor changes have been made to improve 
reproduction quality 

• Points of view or opinions stated »n this docu 
ment do not necessary represent official NIE 
position or policy 



Abstract 



This report Includes two memoranda which were prepared describing 
the results of an evaluation of the use of job aids in training for 
census enumerators. The enumerators were employed in the 1980 census 
to locate and interview households which failed to mail back their 
census questionnaire. 

The first report describes comparisons among participants in two 
training approaches - one using job aids, the other using standard 
reference materials - on a variety of attitudinal and self-report 
measures. The second report compares the performance of the two groups 
of participants on a variety of performance measures collected following 
training. Moreover, recommendations for improving training are also 
included. 

As noted in these memoranda, the data reported have not received the 
review and clearance normally associated with published census documents. 
The reports contain the opinions and recommendations of the author which 
should not be interpreted as positions of the Bureau of the Census. 
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SUMMARY 



Purpose 

In response to criticisms of verbatim training, the Alternative Training 
Experiment (ATE) evaluated the effectiveness of a training approach for 
followup 1 enumerators - called Job-Performance-Aided Training - that re- 
quired changes in both the process used to prepare training materials and 
the methods of instruction used to present those materials. 

Unlike standard census training, the preparation of Job- Performance-Aided 
(JPA) training required that the same person prepare the enumerator field 
reference manual and accompanying training guide to accomplish the following 
specific objectives: 

1. Exclude irrelevant or redundant information from the enumerator 
manual and, when possible, consolidate procedures. 

2. Simplify the manual and the presentation of procedural information 
to improve access and comprehensib-;i ity. 

3. Modify verbatim training so that known advantages could be realized, 
while introducing a variety of learning strategies to simplify 
training, increase trainee participation, and guarantee competency 
on a basic set of defined skills. 

The overriding objective of this research, however, was to evaluate an 
approach to training which was low cost and could be easily adapted to 
known operational constraints in Bureau censuses and surveys. 

Method 

Three pairs of census decentralized district offices were matched on variables 
related to the difficulty of enumeration. Each office in a pair was then 
randomly assigned to one of the two training methods compared in this study; 
i.e., JPA or standard (control) census training. Measures of enumerator 
attitudes (reactions) toward the training were collected anonymously on two v 
occasions; immediately after the conclusion of training and after 2-3 days of 
job experience. Standard census administrative records (e.g., Record of First 
Review, Record of Reinterview, Quality Control Enumerator Daily Progress Record, 
Employee Pay Voucher, and Crew Leader Record of Progress) were used to collect 
and develop enumerator performance and production indices. 

Alternative (JPA) training materials were developed within the Bureau using 
' existing facilities and distributed to participating district offices through 
the Jeffersonville processing center. A more detailed explanation of how JPA 
materials differed from standard enumerator-t raining materials is presented 
in the body of this paper, but, in general, ,the JPA training differed most 
notably in the process used to prepare materials, the followup 1 enumerator 
manual, the nature of activities used during training, stylistic variations 
attributable to presentation style (but not to a given training approach), 
and packaging of the materials. 



Results 

Only the reactions of enumerators on attitudinal measures are discussed in 
this memorandum, plus some supplemental findings dealing with an enumerator's 
fear of working in certain areas and suggestions for improving the enumerator's 
job. Production and performance indices, as well as recommendations for 
improving Bureau training, will be discussed in a followup memorandum. 

The results indicated that lar^e proportions of enumerators in both training 
approaches gave high ntings to their training on a variety of questions 
dealing with quality of the training, feelings of preparedness to perform 
enumerator activities, reading ease of training materials, and the adequacy 
of coverage of specific enumerator tasks during training. In itself, this 
finding contradicts widespread feelings that verbatim training is inherently 
disliked by participants, although it is necessary to caution that the control 
training used in this study may not be representative of verbatim training 
used in other census operations, and the high ratings might be influenced by 
factors other than training quality. In general, although the differences \ 
found were generally small as assessed by attitudinal measures, they 
consistently favored the JPA training and became more pronounced after 
job experience. 



Overview of Paper 

This paper presents a detailed discussion of the development and implementation 
of the Alternative Training Experiment. Accordingly, it was written for a 
diverse audience with disparate interests. To aid the reader in deciding 
which sections are most relevant, the following topical outline is presented: 



Topic Page 



I. BACKGROUND 1 

a. Census Training 1 

b. Alternative Training Approaches: Material Development 2 

and Instructional Method 

c. Job-Perfo nuance -Aided (JPA) Training 3 

d. Hypotheses 4 
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Toptc Page 

II. METHOD 4 

a. Selection of Job Area 5 

b. txperi mental Design 5 

c. Training Material Development 6 

d. Process of Preparing Training Materials: JPA vs. Standard 6 

e. Job Aid Manual 8 

f. Differences in Training Content, Stylistic Variations, 10 

and Packaging 

g. Data Collection Procedures 11 

h. End-of -Training Questionnaire 11 

i. Post-Training Questionnaire 12 
j. Enumerator Performance Record 13 
k. Data Collection Problems 13 

III. RESULTS 15 

a. Percentage of Enumerators Completing Followup 1 15 

Enumerator Training 

b. Enumerators' Self-Ratings of Degree of Preparedness 16 

c. Enumerators' Ratings of Training Adequacy in the 16A 

1970 Enumerator Variance Study 

d. Ratings of Degree of Preparedness Based on Prior 17A 

Census Experience 

e. Enumerators' Ratings of Quality of Training 18 

f. Ratings of Quality of Training Based on Prior 18A 

Census Experience 

g. Enumerators 1 Ratings of Training Materials 19 

Comprehensibil ity 

h. Enumerators' Ratings of the Handbook (Manual) in 19A 

the 1970 Enumerator Variance Study 

i. Enumerators' Ratings of Length of Training 20 
j. Enumerators' Ratings of Length of Training in the 20 

1970 Enumerator Variance Study 

k. Enumerator-Identified "Problem" Job Tasks 21 

1. Enumerators' Reporting Prior Experience 23 

Working in the 1980 Census 

m. Enumerators' Willingness to Work in Other 23 

Census Jobs 

n. Major Dislikes About Training Cited by Enumerators 24 

o. Enumerators' Reported Fear of Working in Certain 25 

Neighborhoods in Their Assignment Areas 

p. Rankings of Suggested Procedures to Follow When an 25 

Enumerator is Afraid to Work in a Neighborhood in 

His/Her Assignment Area 

q. Rankings of Sugqested Improvements (to improve the 26 

enumerator's' job) Given "Most Important" Rating 

IV. DISCUSSION 27 
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Background 

Census verbatim training, which requires a trainer to read word for 
word from a training quide, has been criticized in the past for being 
monotonous and insensitive to the needs of different groups of trainees 
and localities. Moreover, since verbatim training typically uses a 
great deal of lecture, it is open to the criticism that presenting 
a lecture to a group cf trainees is no guarantee that they will either 
comprehend the lecture or be able to follow prescribed procedures once 
they begin working. 

Recognizing these shortcomings of verbatim training, an effort was 
initiated in the Fall of 1978 to develop alternative methods of training 
census employees that would deal with the criticisms lodged against the 
use of verbatim training guides. As will be described more fully in later 
sections of this paper, a variety of alternative training approaches were 
considered, but only one was developed for experimental use in the 1980 Census. 
This research effort was one of six experimental programs that were conducted 
as part of the 1980 Decennial Census Evaluation and Research program. 

Census Training 

Census training is unique in many ways because it is constrained by 
factors not commonly found in most training settings. A list of these 
factors follows: 

1. Most census jobs are short-term (2-4 weeks), which argues against 
the use of expensive or lengthy training sessions. 

2. Applicants for census jobs vary widely in age, education, and 
experience; however, it is assumed thr weak educational skills 
are the norm. 

Note: The data in this report are preliminary and tentative in nature. Users 
of the results memoranda should understand that these documents are prepared *or 
internal office use, with the aim of circulating information among Bureau staff 
members as quickly as possible. These memoranda, therefore, do not undergo the 
careful review and clearance normally associated with published census documents. 
Conclusions and recommendations contained herein essentially reflect the thoughts 
of certain staff members at a point in time, and should not be interpreted as state- 
ments of Bureau position. 



3. Much of census training (e.g., for enumerators) is conducted in small 
groups (8-12 people), in marginal training facilities, and in literally 
thousands of training sites. 

4. Training almost always conducted by i nexperienced trainers who 
received their training only 1-2 weeks prior to training sessions 
that they would lead. 

5. Job manuals are an important reference source. Training is not 
designed so that workers will be able to perform all tasks on the 
basis of recal 1 al one. 



In the context of these previous constraints, the use of a verbatim 
training guide has several advantages. First, a verbatim guide helps 
ensure that the same information is presented to all trainees. Second, 
since the training quide is to be read word for word, inexperienced 
trainers can conduct the training session, assuming only that they can 
read intelligibly to a group of people. Third, verbatim training is a 
low-cost approach since it relies primarily on paper and pencil. Moreover, 
verbatim training writers are not required to have technical skills that 
might be required with the use of more sophisticated audiovisual or 
self-paced training techniques, although learning to write training that 
will be presented orally is a difficult skill to master. Finally, verbatim 
guides and accompanying materials pose few logistical problems in preparation 
or distribution. Certain types of audiovisual s , for example, might 
require long development times, be extremely costly if revisions are 
required, and require training rooms with specialized electrical outlets. 

In the context of census training, other training alternatives have a 
difficult time matching the benefits of verbatim training, particularly 
in terms of costs. Accordingly, although a variety of instructional 
methods were considered as possible candidates for delivering census 
training, serious consideration was given only to approaches that 
emphasized the use of a "paper and pencil" technology and which, if 
successful in an experimental study, had a high probability of being 
used in future Bureau surveys or censuses. 



Alternative Training Approaches: Material Development and 
In structional Method 

When comparing training approaches, a distinction should be made between 
the process used to prepare training materials and the instructional 
methods used to present them. For example, verbatim training is an 
instructional method which may, or may not, use other instructional 
methods, such as audiovisual s, television, audiotape, group discussions, 
role playing, etc. Moreover, t+iere are a variety of competing models 
or approaches which could be used to develop the materials used in a 
verbatim training guide. Needless to say, if either the training materials 
development model or instructional method was faulty, ineffective training 
would likely result. 



3. 



Training materials for the 1980 Census were prepared using what has generally 
been called the Instructional Systems Resign (ISO) model. This model 
requires the following steps: (1) task analysis, (2) specification of 
training objectives, (3) design of instruction and development of learning 
activities, and (4) evaluation of the effectiveness of the training. 
The ISD model is currently the most respected and widely used model of 
instructional design because it introduces standardization and quality 
control into a process that, historically, has been almost totally subjective. 
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Job-Performance-Aided (JPA) Training 

In the military services and industry, job reference manuals are an 
important source of information for successful job performance. Despite 
their importance, prior to the early 1950's relatively little attention 
had been paid to their design and the problems users encountered trying 
to access and comprehend procedural information. Over the past two 
decades, this problem has been studied, primarily by researchers in 
military technical training, who were interested in developing alternative 
formats and presentation methods for the paragraphs of prose descriptions 
so commonly found in technical manuals. 

Initial work in this area focused on the development of job aids, which 
were defined simply as any device that helps a person perform a job 
task. Accordingly, although any manual is, itself, a job aid, develop- 
mental work emphasized the dual problems of access to and comprehen- 
sibility of procedural information. Psychological research discovered 
that varying formats - such as checklists, illustrations, schematic 
drawings, decision trees, algorithms (flow charts), and decision tables - 
improved human performance when compared to standard passages of prose 
materials. However, job aids are not limited to paper-and-pencil 
approaches. Microfiche, computers, and other devices have also been 
designed to aid performance in job tasks. 

The theoretical rationale underlying the use of job aids is that they 
serve as repositories for procedural and factual information that 
does not have to be memorized by the trainee and, hence, covered extensively 
in training. Accordingly, training is simplified and usually shortened. 
Moreover, there is research evidence that job aids help less experienced 
personnel perform at levels comparable to the performance of more 
experienced workers. 

Although job aids are the end product, they represent only the "tip of 
the iceberg" of the process used to produce them. Moreover, job aids 
can be, and have been, used with a variety of training methods. 

The process of developing Job-Performance-Aided (JPA) training uses 
the ISD approach described earlier, but includes one critical improvement. 
Specifically, the JPA approach requires that the design of job reference 
manuals and training proceed concurrently. Probably the most irrniediate 
benefit of this requirement is that subject -matter experts and training 
designers must work together to identify job incumbwrt skills, 
clarify ambiguities in existing or proposed procedures, and to change 

JO 



procedures if limitations in training resources (time, money, or equipment) 
argue against their use. Other benefits include simpler manuals because 
some material that is covered in training can be excluded from the manual 
and simplified training. Training is simplified because a manual employing 
job aids can be used as a quide in group-learning exercises or as a self- 
instructional tool either in the classroom or in a sel f-instructior.al package. 

Available research indicates that job aids and, concomitantly, JPA training 
are suitable for job areas where there are long and complex behavior 
sequences, many tasks exist that are rarely performed, "look-up" tables or 
charts are required, task performance benefits from the use of illustrations, 
and tasks can be mentally rehearsed before they are attempted. Census 
training, which emphasizes the use of manuals both in training and on the 
job and prescribes sometimes complex procedures to ensure standardized 
data collection, appears to be a suitable candidate for the extensive 
use of job aids. 



Hypotheses 

Based on previous research findings that studied the use of job aids, the 
experimental use of job aids in census training should result in the following 

1. Shorter, more simplified training. 

2. Training quality at least equivalent to standard census 
traini ng. 

3. Training materials that are easier to use and comprehend. 

4. Increased use of job reference manuals on the job immediately 
following training. (Only the period immediately following 
training is of interest. After this time, job experience and 
supervisory feedback should 'dramatical ly reduce the frequency 
of manual use. ) 

5. Better job performance by workers exposed to JPA training. 

6. Absence of job performance differences between workers with 
and without prior experience working in the 1980 Census. 

This memorandum will provide data relevant only to the first four hypotheses. 
A later memorandum will address the last two hypotheses and provide* other 
performance data of interest. 



Method 



Initially, two alternative training methods were devised for comparison 
with standard census training. The second approach, called "abbreviated" 
training, replaced the verbatim training guide with a checklist of training 
activities and was planned to take a maximum of six training hours, Like 
the JPA training mentioned earlier, the abbreviated training also used 
job aids. However, only development of the JPA training with verbatim guides 
was continued because of poor trainee performance in a test and lack of 
time to revise the approach. 



5. 



Selection of Job Area 

Although a variety of census jobs were considered for the experimental test 
of JPA training, the followup 1 enumerator position was selected for the 
following reasons. It was the largest single job category in the census 
(preliminary estimates were thai 150,000 persons would work as followup 1 
enumerators); the followup 1 enumerator was required to use job reference 
manuals both in training and on the job; and the job involved numerous 
procedures, some of which were expected to be used only infrequently. 

Experimental Design 

The Ideal design for this study would have been to use both training 
approaches (JPA and standard) in the same district office, resulting in a 
factorial design with training approach fully crossed with district 
office. Unfortunately, this design would have required random assignment 
of enumerators within a DO to training approach, two different sets of 
followup 1 enumerator training materials, two different sets of followup 
1 crew leader training materials, and supervisors (i.e., Field Operations 
Assistants) to become familiar with both training approaches. Considering 
the hectic nature of census operations in DO's and the difficulty of 
controlling field experiments, it was decided that such a design was 
impractical . 

Instead, using available demographic characteristics obtained in the 
1970 Census, three pairs of decentralized census district offices were 
matched on variables thought to be related to the difficulty of enumeration 
(e.g., density of population, educational and income characteristics, etc. - 
see Appendix A for a comparison of the matched pairs of offices). The 
offices in each pair were then randomly assigned to one of the two 
training methods, resulting in three experimental and three control offices. 
Only one training approach was used in a given office. Moreover, only six 
DO's were included so that project control could be maximized. 

Computations of power for this design (DO's nested under two training 
approaches), were calculated assuming that it was desirable to detect a 
difference of 0.05 in proportions for a dependent variable. These calculations 
indicated that there was a 96% probability of detecting a real difference 
between the training groups. 

Only one type of office (decentralized) was included in the sample because 
followup 1 enumerator training differed only slightly between centralized 
and decentralized offices, and concern was expressed that centralized offices 
typically experienced operational problems which could contaminate any research 
fi ndi ngs. 
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6. 



Training Material Developmen t 



The distinction made earlier, between the process used to develop 
instructional materials and the method used to present those materials, 
is a critical one when comparing the experimental JPA training developed 
for this study with the standard (control) census training used for 
decentralized, followup 1 enumerators. 

Both the standard and JPA training used verbatim guides and other instructional 
methods which it was felt would add variety, as well as accomplish specific 
objectives. For example, both training approaches used review exercises or 
performance checks throughout the training, workbooks, practice interviewing, 
training aids, other types of small group interactions, and the samo set of 
audiovisuals. Moreover, the preparer of the JPA training and writers of the 
standard decennial training collaborated with each other and exchanged Ideas 
about potentially effective training activities. 

As a result of this exchange, the JPA training differed most significantly 
from the standard census training in: 

1. The process used to design and develop training materials. 

2. The enumerator job reference manual (the followup 1 enumerator 
manual was redesigned as a job aid manual). 

3. The nature of the activities used during training (since activities 
using the job aid manual differed from comparable activities 
using the standard manual). 

4. Stylistic variations attributable to different writers, not to 
a given training approach. 

5. Packaging of the training materials (e.g., manuals, workbook, 
training aids, census forms, etc.). 



Process of Preparing Training Materials: JPA vs Standard 



The preparation of standard materials for followup 1 enumerator training 
can be summarized as follows: 

1. Procedures were specified by subject-matter and operational divisions. 

2. A manual writer in Field Division prepared a draft of a followup 1 
enumerato r manual . 

3. Drafts of this manual were circulated to subject-matter and operational 
divisions and decennial training guide writers for comments. 

4. Decennial writer(s), who were preparing verbatim training guides 
for followup 1 enumerators, worked from drafts of the manual or 
the final version and used the ISD model to design and develop 
thei r materials. 
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7. 



This process can be contrasted with the following which was used to produce 
the JPA training: 

1. Procedures were specified by subject-matter and operational divisions. 

2. A manual writer in Field Division prepared a draft of a followup 1 
enumerator manual . 

3. Drafts of this manual were circulated to subject-matter and operational 
divisions and the JPA training developer. 

4. The JPA training writer redesigned the followup 1 enumerator 
manual and disseminated it for com. ents within the Bureau. 

5. As the job aid manual was being developed, the JPA verbatim 
training guide for followup 1 enumerator training was also under 
development using the JPA model of instructional design. 



Although step 2 above was included in the JPA process, it was only necessary 
because of the late starting date of this study. If the JPA process were 
to be implemented on a large scale in the Bureau, subject-matter experts 
would deliver their procedures directly to the job aid manual designer. 
Moreover, although one person designed both the job aid manual and the 
JPA verbatim training guide, it is conceivable that these tasks could be 
handled by more than one person if a formal planning process was implemented 
to insure communication between the involved parties. 

Step 5 in the JPA process was especially critical because the concurrent 
development of the training guide and manual meant that "dry runs" or 
demonstrations of the training could generate suggested improvements in 
both the manual and training activities. By the time standard training 
dry runs" were conducted, the design of the enumerator manual was complete 
and no substantive changes could be made. 

Particularly with respect to the manual, it was important to determine if: 

(1) Material could be accessed in a reasonable period of time? 

(2) Once accessed, could it be comprehended? 

(3) Did correct performance result if the material was comprehended? 



Thl; third criterion, although apparently ridiculous (If people can tell you 
how to do something, doesn't that mean they can do it?), was necessary 
because only properly designed training activities — e.g., practice 
interviewing, completing forms, role playing through the steps of an 
interview, solving problems, etc. — could demonstrate if, in fact, a 
trainee was able to successfully apply knowledge obtained in a manual. 
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Another benefit of working with draft copies of both the enumerator manual 
and verbatim training guide during "dry runs" was that unusally complicated 
or unrealistic procedures could be identified. Accordingly, different 
formats for presenting information in the manual could be used, or different 
training activities tried to insure that trainees left training with 
requisite skills and knowledge. Moreover, procedural changes, although 
bothersome, did not cause major problems because an actempt was made to 
modularize the JPA training; that is, major skills or concepts were taught 
using the following approach: introduce the topic, explain and demonstrate 
proper behavior, provide learning exercises, and provide feedback on 
perfo rma nee. 

In summary, the JPA process required that the design of the enumerator 
manual and verbatim training guide complement each other. Moreover, both 
the manual and training guide were tested, and revised as necessary, prior 
to use in the field. Although the standard enumerator training was also 
tested prior to use in the field, opportunities did not exist to change the 
enumerator manual even if it was discovered that trainees could not use 
it for certain tasks. Instead, the burden fell on the decennial writer 
to work around design deficiencies in the manual. In part, changes in 
the manual could not be made because of lack of time, but the manual 
review process also did not give decennial writers the authority to 
implement changes in the manuals which they thought would improve the 
quality and effectiveness of training. 

Job Aid Manual 

Preparation of the job aid manual started with a decision concerning 
which training objectives would be covered solely in training, in the 
manual, or in both places. 

Although they are not mutually exclusive, the following quidelines were 
used. 

Include in training, if the task: 

1. is difficult to learn on the job; 

2. is hard to communicate with words; 

3. requires a great deal of practice; 

4. allows little room for error or results in serious 
consequences if errors are made; 

5. does not require exorbitant sums of money to train; 

6. is performed frequently on the jjb: 

7. does not allow time to reference a manual; 

8. is performed by a large number of people in the job area. 

Include in the manual, if the task: 

1. has long and complex behavior sequences; 

2. is rarely performed; 

3. involves readings and tolerances; 

4. can be mentally rehearsed before the need to perform it arises; 

5. benefits from the presence of illustrations; 

6. requires reference information (e.g., tables, graphs, flow charts 
schematics). 



9. 



Job aids were then developed from prose descriptions of an enumerator's 
duties. Appendix ? (page 32) presents a job aid (flowchart) that was 
developed for the situation where a nonresponse household was occupied, 
but the enumerator discovers that the respondent did not live in the 
housing unit on census day. Page i of Appendix B shows a copy of the 
page of the Table of Contents which the enumerator would have used to 
loctte this information. The enumerator would have first looked in 
Section 3 (A Housing Unit Is Occupied) and, then, Section 31 for the 
relevant procedure to follow. 

Page iii of Arcendix B shows the equivalent page from the standard enumerator 
manual which would have been used to locate the procedure for this situation. 
In this case, the followup enumerator would first have had to look at 
Section 7 (How to Enumerate Nonresponse Cases) and, then, Section 7.B 
(Determining the Status of the Housing Unit). The specific situation 
causing problems for the enumerator is not directly identified in the 
Table of Contents. 

Pages 52 and 53 of Appendix B are copies of the relevant section of 7.B. 
Subsection 7.Bl.b (Housing Unit Occupied Entirely by Persons Who Did Not 
Live There on Census Day) describes the procedures to follow. 

This comparison points out several distinguishing characteristics of a 
job aid manual. For example: 

1. Problem situations encountered during field work can be readily 
identified in the Table of Contents, hence access to procedural 
information is improved. 

2. Steps in a procedure are clearly identified, decision points 
are clear, and end products specified. 

3. Steps in a procedure are presented in the same sequence that 
would be followed on the job. 

4. Clear writing is emphasized.' Action verbs are used, jargon is 
avoided, highlighting is used. 
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Using job aids to present procedural information, and excluding material 
from the manual which was covered in training, resulted in a 62-page 
JPA manual versus the 129-page standard manual* 



Differences in Training Content, Stylistic Variations, and Packaging 

Tests or "d^y runs" of the JPA training guide indicated that it required 
approximately 25% less lecture time than the standard enumerator training, 
even though comparable material was covered* Further, since followup 1 
enumerators were told that they would be paid for two days of training, 
it was decided to lengthen the JPA training by adding more workbook 
activities (approximately twice as many workbook exercises were used in t' e 
JPA training than in the standard training) and an end-of-trai ni ng, remedial 
session for enumerators who failed to obtain a certain score on the JPA 
final -review exercise. Unlike the final review exercise used in the 
standard training, which was entirely multiple choice, the JPA final-review 
exercise required enumerators to fill out address labels, address listing 
pages, and M For Census Use Only" boxes from the questionnaires; use the 
job aid manual to solve possible field problems; and answer multiple-choice 
items. If an enumerator got more than a predetermined number of errors on 
this final exercise, the trainer was instructed to give the enumerator 
compensatory (remedial) classroom instruction. Compensatory classroom instruc- 
tion was not a feature of the standard training. However, on-the-job training 
for enumerators who performed poorly during classroom training remained a part 
of both the JPA and control training. 

Stylistic variations between the JPA and standard training guides reflected 
differences that tend to be unique to individual writers. The JPA 
verbatim guide was written in a more informal, colloquial manner and used 
significantly more trainer-directed questions than the standard guide 
(one question asked, on the average, every 176 words for the JPA guide 
v« one question every 293 words for the standard guide). 

Finally, the JPA training packaged all tha requisite training mate Mais 
differently. Rather than assemble all the materials (workbook, manuals, 
training aids, blank forms, supplies) into a single large manila envelope, 
as was done in the standard training, the JPA materials were packaged in 
expandable portfolios. Further, each portfolio was subdivided into 13 
sections, each with a simple name and numerical identifier. These names 
and numbers facilitated access to specific materials during the training 
and helped to lessen trainee feelings of disorganization and confusion 
so common at the start of any training session. Moreover, frequently 
used materials (e.g., manual, workbook, and certain training aids) were 
color-coded so that the trainer could say M Turn to page 10 in the green 
manual", instead of "Turn to page 10 in the Followup 1 Enumerator Manual, 
Form D-548". 

It was estimated, assuming a production order of 100,000 training kits, that 
these packaging modifications increased the cost of each enumerators training 
kit by approximately $1.40. This figure, however, does not include the cost 
of assembling the kits at a processing center. It is estimated that assembly 
t^nrie would double because of the sequencing required for the training materials. 
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Data Collection Procedures 

Since this study was conducted in six different geographical locations, a 
full-time, data-capture clerk was trained in each office to coordinate the 
study, distribute and collect evaluation instruments, and deal with any 
problems that occurred. All of these individuals had previous census 
experience. 

Two anonymous attitude que&tlonnai res — one given at the conclusion of 
training; the other after 2-3 days of work experience ~ and an 
enumerator performance record were the major evaluation instruments. 
Each of these will be discussed more fully in the following sections. 

Anonymous questionnaires were used because it was conjectured that enumer- 
ators would give more valid responses, particularly for questions that 
indirectly or directly reflected on the competence of their supervisor 
who was responsible for distributing and collecting the questionnaires. 

Demographic data was not requested because it was felt that enumerators 
would not understand the reason such information was needed and, further, 
certain demographic variables (age and sex) cannot be legally used to - 
select census enumerators for training purposes. 



End-of-Training Questionnaire 

A questionnaire was completed by each enumerator immediately at the con- 
clusion of the training session to measure attitudes toward the quality 
and effectiveness of the training (See Appendix C). In addition, questions 
were asked about prior census experience (item 1, Appendix C) and type of 
training setting (item 3, Appendix C) so that prior experience could be 
controlled for in later analyses, and to determine the extent and length 
of on-the-job training that occurred. 

On-the job-training (OJT) for replacement enumerators was of special 
interest because in some district offices it was expected that replacement 
enumerators would comprise a large proportion of the work force and, in 
the JPA group, a more intensive type of on-the-job training was used. 
It was planned to be longer than the comparable training used in the 
control group and used more formal learning activities. Therefore, a 
comparison of enumerator reactions and performance was of interest. 
Unfortunately, although crew leaders were instructed to have replacement 
enumerator; complete the end-of-traini ng questionnaire after OJT, this 
instruction was followed so rarely that no meaningful analyses were 
possible. 

Specific questions about preparedness, training length, quality, and 
reading ease of training materials were asked because each of these 
features could possibly reflect differences between JPA and standard 
census training. A more detailed question about specific parts of the 
enumerator job (item 8, Appendix C) was asked to determine if the training 
approaches differed in their adequacy of coverage of critical job skills. 
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A question about the value of different training activities in helping a 
trainee learn the enumerator job was asked to determine if the activities 
stressed in the JPA training (e.g., using the job aid manual and workbook) 
were actually perceived that way by trainees. Moreover, it was hoped 
that this question would provide feedback on the relative utility of 
different activities within a training approach. For example, some 
training specialists have argued for an increased use of audiovisuals 
in Bureau training. It was of interest to determine if audiovisuals 
were used more effectively in one training approach, as opposed to the 
other. 



Post-Training Questionnaire 

The post-training questionnaire (see Appendix D) was to be completed by 
each enumerator during "First Review" or earlier, if the enumerator was 
released for any reason. First Review was usually scheduled to occur 
2-3 days after training and was a crew leader's initial examination of an 
enumerator's work. It included such things as reviewing entries in the 
Master Address Registers and edits of a sample of long and short census 
questionnaires. 

First Review was also the point beyond which performance differences 
between training approaches were expected to wash out because, if an 
enumerator's performance was found to be inadequate, the crew leader 
was supposed to qive the enumerator on-the- job-training to deal with 
the problems. 

The primary purpose of the post-training questionnaire was to determine 
if an enumerator's work experience significantly changed attitudes obtained 
on the end-of-training questionnaire. For example, an enumerator might 
have rated a training approach highly immediately after training, but 
changed his/her opinion after encountering the realises of the job. 
In addition, a question (item 8, Appendix D) was also asked about the 
frequency of use of the enumerator manual (or job aid manual) and the 
Questionnaire Reference Book (QRB). Although the QRB was not redesigned 
to be a job aid manual, it was still considered of interest to determine 
how often the QRB was used. Also, since obtaining observational data to 
estimate the frequency of manual use was not practical, a self-reported 
answer was the only way available to obtain these data. 

Other questions (items 6 and 7) on the post-training questionnaire 
attempted to measure indirectly the effectiveness of the JPA training. 
Item 10 dealt with a list of suggested improvements that enumerators 
were asked to respond to. And item 11 dealt with an issue that may 
affect data-collection procedures in future censuses and surveys; that 
is, enumerator fear of working in certain neighborhoods. Finally, item 12 
sought to determine the employment status of census employees in the 
participating district offices. Again, this information might be of 
importance in planning future censuses. 
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Enumerator Performance Record 

Data-capture clerks in the district offices were instructed to complete a 
performance record (see Appendix E) for each person trained to be a 
followup 1 enumerator. The data from this form will be reported in the 
next memorandum in this PERM series. 

Data Collection Problems 

Training was scheduled to be conducted at the same time (April 14-15, 1980) 
for all enumerators, but unexpected field problems caused delays of as 
much as a week in certain offices (see Appendix A for a detailed breakdown 
of training dates for each district office). 

Since the primary job of a followup 1 enumerator was to obtain census 
information for households that failed to mail back a census questionnaire, 
this particular position was expected to last only 3-4 weeks. In some 
offices (New York City area), however, this time period approached three 
months. 

All enumerators were also supposed to be paid piece-rate for each questionnaire 
completed, but field problems resulted in all offices paying some of their 
enumerators an hourly wage instead. A variation in pay is important because 
it could affect attrition and attitudes toward the enumerator job, regardless 
of training. 

Table 1 shows the percentages of end-of-traini ng and post-training questionnaires 
(1005 sample was used) that were returned from the JPA and control (standard 
training) offices. 



TABLE 1 

Percentage of Evaluation Instruments Returned 



JPA CONTROL 
97.6 79.5 

1/ 



77.9 40.3 

1/ 



i/ An unknown number of these forms were reportedly lost in the mail 



Despite the fact that some instruments were lost in a mail transmittal from 
one control office (a maximum of 15% of the total number of forms in the 
control group were lost) return rates for all the evaluation measures 
were still lower in the control offices. Although it is not possible to 
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Post-training (after 2-3 days of job 
experience) attitude questionnaire 
(D-860) 
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say with certainty why these differences occurred, interactions with the 
data-collection clerks indicated that clerks in the JPA offices were more 
successful in establishing good working relationships with field personnel 
so that materials were more efficiently distributed and returned. 

One possible explanation for better worki ng relationships was simply 
better clerks, but there were also indications that a "Hawthorne" effect 
was operating for supervisors. Field Operations Assistants (FOA's) in 
the JPA offices trained both with standard census materials and the JPA 
materials, whereas FOA's in the control offices saw no real differences 
in procedures, except for the added work of additional evaluation forms. 
Further, FOA's from one v PA office who were asked to compare the JPA and 
standard training approaches expressed a decided preference for the JPA 
training. Therefore, although all participating offices were visited by 
Bureau observers and knew that they were part of an experimental study, 
supervisors in the JPA offices might have exerted more effort in helping 
the data-collection clerks collect evaluation measures. 

If, in fact, the preceding did occur for FOA's in the JPA offices, there 
may be reason to suspect systematic bias in the return of the post-training 
attitude questionnaire and administrative records used to complete the 
enumerator performance record. This bias would result from FOA's exertina 
more effort to obtain forms from possibly marginal or unmotivated crew 
leaders. Unmotivated crew leaders could conceivably have a detrimental 
effect on both enumerator attitudes and performance. Conversely, motivated 
crew leaders who conscientiously performed the^'r duties and returned their 
evaluation and administrative records, could be expected to influence 
positively their crews and production measures. 

Although the possibility of a systematic bias cannot be discounted, 
particularly for the post-training questionnaire and enumerator perfor- 
mance record, it does not seem likely that a "Hawthorne" effect, or other 
systematic bias, affected the results of the end-of-trai ni ng enumerator 
questionnaire. Enumerators (except those with prior census experience) 
saw only one type of training and completed the questionnaire immediately 
after training. Moreover, no special attention was paid to enumerators in 
either the JPA or control offices. 

Another possible confounding factor in t M s study was the presence of 
unexpected, and severe, field problems in the district offices. Field 
problems (particularly problems with address registers) were a major 
cause for concern in all the district offices participating in this study 
but, again, it cannot be said with certainty whether they were worse in 
one type of office, as opposed to the other. This issue will be discussed 
more fully later in this paper and in the second memorandum <n this PERM 
series which will discuss performance measures. 
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RESULTS 

One view of the importance of enumerator training is that it affects an 
enumerator's attitudes toward the census and his/her job and, therefore, 
might contribute to poor performance or attrition. It has even been 
conjectured that training quality might affect attrition rates during 
trai ning. 

Table 2 below shows that the training groups compared in this study did 
not have significantly different rates of enumerator attrition during 
training. 



TABLE 2 

Percentage of Enumerators Completing Followup 1 Enumerator Training 







(N-1,197) 


(N=l,389) 






JPA 


Control 


Finish 


Yes 


98.3 


97.1 


Training? 


No 


1.7 


2.9 



A critical outcome of any training is how prepared a trainee feels to 
perform assigned duties. Low levels of self-confidence existing at the 
start of a job might result in poor performance, more supervision, 
more on-the-job-t rai ni ng, or higher rates of attrition. Further, although 
feelings of preparedness at the end of training are important, such 
feelings after job experience are a more valid measure of the training's 
effectiveness since the enumerator's concrete experiences provide a more* 
valid basis for judging the adequacy of training. \ 

Table 3 presents the responses of enumerators to questions about preparedness 
on the end-of-traini ng and post-training questionnaires. In this analyses 
and subsequent ones comparing end-of-trai ni ng with post-training responses, 
it was assumed that missini questionnaires were randomly distributed in both 
groups and that field problems or other factors (e.g., supervision) did not 
differ significantly between the groups. 
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TABLE 3 

Enumerators 1 Self-Ratings of Degree of Preparedness 

Question: "How well prepared do you feel (were you) to go out and work 
as an enumerator?" 







End-of-training 


Post-traini ng 






a. 


b. 


c. 


d. 




N-1,158 


N«l,085 


N«919 


N=557 




JPA 


Control 


JPA 


Control 


If il 1 Prepared 


47.8% 


48.2% 


60.8% 


54.8% 


Adequately prepared 


51.6 


50.1 


38.0 


42.7 


Poorly prepa red 


0.6 


1.6 


1.0 


2.3 


Not prepared 


0.0 


0.1 


0.2 


0.2 



Most enumerators in both training approaches, and on both occasions, reported 
that they had been "adequately" or "well prepared" for their jobs. The 
marginal association between degree of preparedness and type of training 
at the end-of-training (i.e., between columns a. and b.) was not significant 
( 6.15, 3 df, p * 0.1), but it was significant ( x 2 » 8.45, 3 df, p = 0.04) 
for the post- training comparison (i.e., between columns c. and d.). 

To test whether or not reported feelings of preparedness changed dif farential 1y 
between training approaches after job experience, the partial association of 
factors in Table 3 were tested using a log-linear model analysis available 
in the Biomedical Computer Proqrams P- Series (BMDP-77) statistical package.I/ 
To do this analysis, random patterns of nonresponse were assumed for both 
the end-of-training and post-training questionnaire. Further, it was assumed 
that field problems and supervisory practices did not differ between the JPA 
and control groups. 

The best-fit model included all 2-way interactions, but not the 3-way 
interaction. The presence of a 3-way interaction would have indicated that 
response patterns for the JPA and control groups differed depending on 
the time of measurement. Or, in other words, job experience would have 
significantly changed the response patterns found on the end-of-training 
questionnai re. 




U Brown, M.B. (Ed.) BMDP Biomedical Computer Programs P-Series 1977 , 
Berkeley, CA: University of California Press, 1977. 
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A similar question about feelings of job preparedness was asked on an attitude 
questionneire used in the 1970 Enumerator Variance Study.I/ Although a sample 
of 1,058 decentralized followup enumerators completed this questionnaire, 
differences in training content (followup enumerators performed some different 
tasks in 1970), question wording, and time of administration (it was completed 
when an enumerator either completed his/her assignment or quit) preclude any 
definitive comparisons. Still, the results in Table 3A suggest that both the 
JPA and standard 1980 followup training might have done a better job of preparing 
enumerators for thei r jobs. 



TABLE 3A 

Enumerators' Ratings of Training Adequacy in the 1970 
Enumerator Variance Study 

Question: "How well did you understand your job when you began? 

That is, how adequately did your training prepare you for the job?" 



Response Percent 

Definitely adequate 40.9 

Just about adequate 45.7 

Not quite adequate 10.0 

Definitely not adequate 3.4 



Table 3B in Appendix F shows the responses of the matched pairs of offices 
to the questions about preparedness. Table 3C (Appendix F) presents 
chi-square comparisons for the three pairs of offices. 

For the end-of-trai ni ng questionnaire, the Pattern in Table 3 for the 
"well" and "adequately prepared" categories is reoeated in Table 3B for 
the first two office pairs; however, the percentage differences are larger. 
The third pair runs counter to this trend, with the S. Dayton office giving 
proportionally more "well prepared" ratings. The pattern in Table 3 for 
the "poorly" or "not prepared" categories is the same as that in Table 3B. 

For the post-training questionnaire, the pattern in Table 3 for the "well" 
and "adequately prepared" categories is repeated in Table 3B, with the 
exception of the "adequately prepared" category for the first office pair. 



1/ Inderfurth, Gail. Results from the 1970 Enumerator Variance Study Post- 
Enumeration Questionnaire. E18 No. 40 
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The pattern in Table 3 for the "poorly" or "not prepared" categories 
is repeated in Table 3B for the first two office pairs; however, the third 
pair runs counter to this trend with slightly more S. Dayton enumerators 
reporting that they were "poorly prepared." 

To determine if there was any significant withi n-of f ice variation on the 
preparedness question as a function of experience, the 4-category response 
scale was collapsed into two categories-prepared or not prepared. Table 3D 
in Appendix F presents the results of chi-square tests computed for each 
office. Only the responses in one office (S. Dayton) changed significantly, 
but the change was not of practical significance since 98.1% of the enumerators 
on the post-training questionnaire reported that they felt prepared versus 
99.8% on the end-of-trai ni ng questionnaire. 

It has been argued in some of the research that studied the use of JPA's 
that less experienced personnel using job aids were able to perform at 
levels of proficiency comparable to persons with more job experience. 
Although asking about job preparedness is obviously not the same thing as 
measuring actual performance, it can be conjectured that enumerators who 
differed in prior census experience would not differ in their feelings 
of job preparedness after JPA training, but that they would differ after 
standard census training. The same relationship would also hold for the 
post-training questionnaire results. 

Table 4 presents a breakdown of Table 3 controlling for prior census 
experience. It is apparent in Table 4 that just the opposite of the 
preceding hypotheses was true for JPA-trained enumerators on the end-of- 
traimng questionnaire. That is, comparing columns a. and d. , 60% of 
JPA enumerators with prior census experience felt that they were "well 
prepared" versus 46.1% of the JPA enumerators with no prior experience 
Moreover, no JPA enumerators with prior census experience reported that 
they felt "poorly" or "not prepared". The response patterns in columns 
a. and d. differ significantly (see Table 5). The response differences 
on the post-training questionnaire were not statistically different. 

Again, contrary to the hypothesis for the control enumerators, there were 
no statistically significant differences, controlling for experience, on 
either questionnaire. 

The results in Table 4 suggest that JPA enumerators with prior census 
experience reacted more positively to their training than control enumerators 
with prior census experience. For example, looking at only the "No Prior 
Experience" subsections of Table 4; it is apparent that the pattern of 
results closely approximates that found in Table 3. However, especially for 
the end-of-trai ning comparison, just the opposite 1s true for the "Prior 
Experience" subsections. In contrast to Table 3, proportionally more JPA 
enumerators with prior experience, than control enumerators with prior 
experience, marked the "well prepared" response, although these differences 
were not significant (see Table 6). 
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Ratings of Degree of Preparedness Based on Prior Census Experience 




Question Pre: 


"How well prepared do you feel to go out 


and work 


as an enumerator?" 






Post 


: "How well prepared were you to go out and work as an enumerator?" 










End-of -Training 












Prior Experience 






No Prior Experience 






(145) 
JPA 
a. 


(188) 
Control 
b. 


(333) 
Overall 
c. 


(1013) 
JPA 
d. 


(895) 
Control 
e. 


(1908) 
Overal 1 
f. 


Hell prepared 


60.0% 


53. 2X 


56. 21 


46. IX 


47. 2X 


46. 6X 


Adequately prepared 


40.0 


46.3 


43.5 


53.2 


50.9 


52.1 


Poorly prepared 


0.0 


0.5 


0.3 


0.7 


1.8 


1.2 


Not prepared 


0.0 


0.0 

Prior FxnpMpnrp 


0.0 

• oat" i ra i n i nQ 


is. yj 


n 1 

No Prior Experience 


0.1 




(106) 
JPA 


(105) 
Control 


(211) 
Overal 1 


(813) 
JPA 


(452) 
Cont rol 


(1265) 
Overal 1 


Well prepared 


66.0% 


58. IX 


62. IX 


60. IX 


54. OX 


57. 9X 


Adequately prepared 


32.1 


37. 1 


34.6 


38.7 


44.0 


40.6 


Poorly prepared 


1.9 


3.8 


2.8 


0.9 


2.0 


1.3 


Not prepay 


0.0 


l.O 


0.5 


0.2 


0.0 


0.2 



v 0TEr Base for percentages Is reported 1n parentheses above group designation. 



TABLE 5 

Chi-Square Statistics for JPA and Control Groups Controlling for Prior Census Experience 

End-of-Traininq Post-Training 

2 2 

X p X 

JPA (Cols. a. vs d.) 10.4 .01 2.9 

Control (Cols. b. vs e. ) 3.6 .31 6.7 

Overall (Cols. c. vs f . ) 11.9 .01 6.1 .11 



P 

.41 
.31 



NOTE: All comparisons are based on 3 degrees of freedom. 



TABLE 6 



Chi-Square Statistics for JPA vs Control Groups for End-of-Training and Poit-Traini»;g 



Prior Experience 
2 



End-of-Training 
Post-Training 



X 

2.2 
2.6 



P 

.33 
.45 
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No Prior Experience 
2 



6.5 
7.9 



P 

.09 
.05 



O i 



00 
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Another hypothesis was that JPA training would be more highly rated in 
terms of quality because the enumerator manual and training were designed 
to complement each other, the training was simplified, and more feedback 
(in the form of workbook activities and trainer questions) was provided to 
enumerators so that they cculd concretely evaluate what they did or did 
not know.iy 

Table 7 presents the responses of enumerators to questions about the quality 
of training on the end-of-trai ni ng and post-trai ni ng questionnaires. 



TABLE 7 

Enumerators 1 Ratings of Quality o. Training 





End-of-trai ni ng 




Post-trai ni ng 




a. 

N=l,163 


b. 

N-1,093 


c. 

N=887 


d. 

N=526 




JPA 


Control 


JPA 


Control 


Very Good 
Good 
Fai r 
Poor 


57.0% 
37.8 
4.5 
0.7 


49. OX 
42.1 
8.3 
0.5 


58.7% 
37.1 

3.0 

1.1 


44.9% 
46.6 
8.0 
0.6 



Most enumerators in both training approaches, and on both occasions, reported 
the quality of the training to be "good" or "very good". However, proportionally 
mere "very good" ratings resulted in the JPA group. The marginal association 
between quality and type of training group was significant for both the end-of- 
trai ning ( x^ = 22.7, 3 df, p=0.00) and the post-training comparisons ( x* = 36.8. 
3 df, p -0.00). 

Table 7A in Appendix F shows the response patterns of the matched pairs of 
district offices to the question about quality. Table 7B (Appendix F) presents 
chi-square comparisons for the three pairs of offices. 

For the end-of-trai ni ng questionnaire, the pattern in Table 7 for the "good" 
and "very good" categories holds for the second and third office pairs in 
Table 7A. The first pair runs counter to this trend with the S.W. Brooklyn 
office giving proportionally more "very good" ratings. The pattern in Table / 
for the "fair" and "poor" categories 1s repeated in Table 7A, except for the 
third office pair where there is a slight deviation. 



1/ Feedback about performance has been shown to be a powerful, and positive, 
influence on learning. 

ERIC 



TABLE 8 

Ratings of QuaHty of Training Based on Prior Census Experience 
Question: "Overall, how would you rate the quality of the training?' 



End-of-Tra1n1ng 



Prior Experience 



No Prior Experience 





(145) 
JPA 
a. 


(188) 
Control 
b. 


(333) 
Over al 1 
c. 


(1018) 

.IDA 

urn 

d. 


(903) 
uonc ro i 
e. 


(1921) 
Overal 1 
f. 


Very Good 


62.8% 


47.9% 


54.4% 


56.2% 


49.3% 


52.9% 


Good 


32.4 


44 7 


jj« j 


38.8 


41.5 


40.0 


Fair 


4.8 


7 4 




4.4 


8.5 


6.4 


Poor 


0.0 


0 0 


ft ft 

Post-Training 


0.8 


0.7 


0. 7 






Prior Experience 




No Prior Experience 






(98) 
JPA 


(100) 
Control 


(198) 
Overall 


(784) 
JPA 


(426) 
Control 


(1210) 
Overal I 


Very Good 


69. 4% 


45. OX 


57.1% 


57.4% 


44.8% 


53.0% 


Good 


25.5 


41.0 


33.3 


38.5 


47.9 


41.8 


Fair 


3.1 


13.0 


8.1 


3.1 


6.8 


4.4 


Poor 


2.0 


1.0 


1.5 


1.0 


0.5 


0.8 



NOTE: Base far percentages Is reported 1n parentheses above group designation. 
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Table 9 

CM-Square Statistics for JPA and Control Groups Controlling for Prior Census Experience 
End-of-Tra1n1ng Post-Training 

2 

X p 

6.9 .07 

5.1 .16 

9.2 .03 



2 

X P 



JPA (Cols. a. with d.) 3.4 .33 

Control (Cols. b. with e.) 1.9 .59 
Overall (Cols. c. with f.) 2.6 .46 

NOTE: All comparisons are based on 3 degrees of freedom 

TABLE 10 

Chl-Square Statistics for JPA vs Control Group for End-of-Tra1n1ng and Post-Training 
Prior Experience No Prior Experience 

2 2 
x 2 P x P 

End-of-Tra1n1ng 7.4 .03 18.1 - 00 

Post-Training 15.1 .00 23.9 .00 

NOTE: All comparisons are based on 3 degrees of freedom. 
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For the post-training questionnaire, the patterns for all response cate- 
gories in Table 7 are repeated in Table 7A, except for the first pair where 
there are slight deviations for the "Good" and "Poor" categories. 

To determine if there was any significant within-office variation on the 
question about training quality, the 4-category response scale was collapsed 
into two categories— acceptable and unacceptable. Table 7C in Appendix F 
presents the results of chi-square tests computed for each office. None 
of the comparisons were significant. 

Although there are no findings in the research literature which would 
predict how more experienced personnel would rate the quality of JPA 
training, it can be conjectured that JPA-t rained enumerators with prior 
census experience would give higher ratings of quality to JPA training 
than JPA-trained enumerators without prior experience since they would 
have previous census training as a standard for comparison. This 
outcome should not occur for control enumerators, unless the standard 
(control) training used in this study also deviated qualitatively from 
typical census training packages. 

To test this hypothesis, chi-square statistics were comouted (see iable 9) 
for each group (JPA and control) controlling for prior census experience. 
Neither one of the comparisons was significant at a .05 significance level. 

Again, looking at just those groups with no prior census experience in 
Table 8, the response pattern obtained closely approximates that found 
in Table 7. This is not the case, however, for groups with prior census 
experience. Proportionally more JPA-trained enumerators with prior 
experience gave "very good" ratings to their training than their control 
counterparts with prior experience. Both of these comparisons were 
statistically significant (see Table 10). 

A major argument for using job aids in training and manual design was th*t 
they simplified the presentation of procedural information and improved 
comprehensibility. Accordingly, a question (on the end-of-t raining question- 
naire only) asked whether training materials were easy to read and understand. 
Table 11 presents the responses to this question. 

TABLE 11 

Enumerators 1 Ratings of Training Materials Comprehensibility 



Question: "Were the training materials easy to read and understand?" 





N=l,150 


N« 1,080 




JPA 


Control 


Yes 


95.8% 


90.7% 


No 


4.2 


9.3% 



The training materials in both approaches were judged easy to read and 
understand, but the JPA approach eliciteJ a small, but significantly 
higher rating ( x 2 3 23.4, 1 df, p » .GO). 
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Table 1 1 A presents the responses of enumerators to a similar question that 
was asked on the questionnaire used in the 1970 Enumerator Variance Study. 
That question, however, referred only to the descriptions and directions in 
the enumerator's handbook (manual) so, as before, any differences are only 
suggestive. 



TABLE 11A 

Enumerators' Ratings of Handbook (Manual) in the 1970 
Enumerator Variance Study 

Question: "Did you think the descriptions and directions in the 
enumerator's handbook were--" 



Response 


Pe rcent 


Very clear 


31.1 


Mostly clear 


51.7 


Somewhat confusing 


14.0 


Very confusi ng 


3.2 



One criticism made against the use of verbatim training is that it can 
easily slip into monotony, and trainees frequently complain about training 
sessions that are too long. 

JPA training using verbatim guides was designed specifically to be activity 
oriented, provide a variety of learning activities, and require active parti- 
cipation by each enumerator. Accordingly, it was conjectured that JPA-verbatim 
training would be perceived as more acceptable in length than the control- 
verbatim training. A question that asked directly about the length of train- 
ing was asked on the end-of-trai ni ng questional re and the results are pre- 
sented in Table 12. 
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TABLE 12 

Enumerators' Ratings of Length of Training 



Question: "Was the length of training..." 





N = 1,162 


N = 1,088 




J PA 


Control 


Too Short 


12.lt 


15.1% 


About Right 


81.0 


72.5 


Too Long 


6.9 


12.4 



It is apparent in Table 12 that proportionally more enumerators rated the JPA 
training as acceptable in length ( x 2 ■ 26.8, 2 df, p = 0.00). 

Table 12A presents the results of a similar question that was asked in the 1970 
Enumerator Variance Study. As before— because of differences in training, con- 
tent, time during the followup period that the question was asked, and variations 
in question wo rdi ng--differences in response are only suggestive, but it appears 
that the lengths of both training approaches investigated in this study were more 
acceptable than 1 i 1970. 

TABLE 12A 

Enumerators' Ratings of Length of Training in the 1970 
Enumerator Variance Study 



Question: "Did you think the amount of time you spent in training to 
be an enumerator was" 



Response 


Percent 


Too long 


10.1 


A little too long 


21.9 


Just right 


37.8 


A little too short 


22.8 


Too short 


7.4 



In a previous analysis that dealt with enumerator feelings of preparedness 
before and after job experience, It was conjectured that the self-confidence 
an enumerator felt in approaching a job wa important because it could 
affect performance, the amount of supervision or on-the-job training required, 
and attrition. Although that analysis dealt with general feelings of 
preparedness, it was equally important to determine enumerator self-confidence 
about specific job tasks since these could be linked more directly to 
differences in training approaches. For example, the JPA training stressed 
using the job aid manual and dealing with unusual situations. Moreover, 
it Included more and different learning activities dealing with completing 
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payroll forms, questionnaires, and address registers. Further, 1t was more 
pessimistic about the possible cooperation of respondents during followup 1. 
On the other hand, the control training required more practice interviewing. 
Accordingly, differences between the JPA and control training in enumerator 
perceptions could be expected on these dimensions. 

On the end-of-training and post-training questionnaires, enumerators were 
asked to check tasks that they thought would cause some problems (end-of- 
training) or were not covered well enough (post-training) in training. 
Table 13 lists tne major tasks in an enumerator's job and shows the 
percentage of JPA or control enumerators who checked tasks on either 
questionnai re. 

TABLE 13 

Enumerator-Identified "Problem" Job Tasks 



End-of-Tra 
(Expecting Diff 





N = 1,168 


TASK 


JPA 


Locating addresses 


8.0% 


Interviewing people 


11.6 


Completing the questionnaire 


7.7* 


Completing the address register 


4.6 


Fi'ling out payroll forms 


8.2* 


Using the job reference manuals 


5.1* 


Getting people to cooperate 


49.4* 1 


Dealing with unusual situations 


42.6 


Answering respondent questions 


10.4 


Checking for missed housing units 


19.4 



ining Post-Training 
iculties) (Not covered Well 

Enough in Training) 



N * 1,097 


N = 932 


N = 560 


Control 


JPA 


Control 


6.9% 


8.3% 


9.5% 


9.1 


6.9 


6.8 


5.6* 


5.2* 


8.9* 


6.0 


4.6* 


9.1* 


14.9* 


11.3* 


15.7* 


9.2* 


4.4* 


8.6* 


36.9* 


14.7* 


20.7* 


39.1 


17.8* 


28.0* 


10.8 


6.9 


8.4 


18.6 


12.7 


15.5 



♦Indicates that chi-square statistic (JPA vs Control) is significant, p< .05, 1 df. 



Known differences between the JPA and control training apparently affected 
enumerator perceptions of the tasks that they expected to have difficulty 
with at the conclusion of training. For example, in JPA training, which 
was more pessimistic than the control training about respondent cooperation, 
enumerators expected more difficulty getting people to cooperate. Further, 
they expected fewer problems using job reference manuals and filling out 
payroll forms. However, they expected more problems completing the question- 
naires (i.e., filling in address labels, "For Census Use Only" box, 
back page, population and housing Items).!/ 



1/ On some of the tasks in Table 13, for example, interviewing people and 
dealing with unusual situations (end-of-training), larger percent differ- 
ences between the JPA and control groups were not significant because 
ERIC usin9 the cni " S( l uare statistic, variability was greater. 
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On the post-training questionnaire, however, all these differences favored 
the JPA training, plus dealing with unusual situations and completing the 
address registers became statistically significant differences. 

As with previous analyses, the effect of prior census experience on an 
enumerator's self -confidence in performing job tasks is of interest. It 
can be conjectured that JPA trained enumerators, whether or not they had 
prior census e perience, would not differ in either their expectations of 
expected job problems or their opinions about which tasks needed more 
coverage in training. On the contrary, it was expected that control 
enumerators, with and without prior experience, would differ significantly 
in their reactions because of the absence of job aids to help performance. 

Table 13A in Appendix F presents a breakdown of Table 13 controlling for 
prior experience. Table 13B presents the results of chi-square tests 
that compare enumerators in either group who had prior census experience 
with those who did not (e.g., JPA enumertors with experience vs. JPA 
enumerators without experience). 

The results do not support the hypothesis. Although there was only one 
(checking for missed housing units) significant difference for JPA 
trained enumertors on the end-of-trai ni ng questionnaire, there was also 
only one difference (getting people to cooperate) for the control enumerators. 
Moreover, on the post-training questionnaire, there were two significant 
differences (completing payroll forms and checking for missed housing 
units) for the JPA group, but only one (interviewing people) for the 
control group. 

If the "No Prior Experience" group in Table 13A is compared with Table 13, 
it is apparent that the response pattern is closely approximated. However, 
a different pattern occurs when the "Prior Experience" group 1s examined 
separately. Tabie 13C presents the results of chi-square tests comparing 
the JPA and control groups for different levels of experience. 

JPA and control enumerators with prior experience, and before work 
experience, differed in their perceptions on only one task (filling out 
payroll forms) that they thought would cause problems when they started 
working, whereas JPA and control enumerators with no prior experience 
differed in their perceptions of the expected difficulty of three tasks 
(filling out payroll forms, using job reference manuals, and getting 
people to cooperate). 

After job experience, the JPA and control enumerators with prior experience 
differed on two tasks (filling out payroll forms and dealing with unusual 
situations), whereas the JPA and control enumerators with no prior experi- 
ence differed on six tasks (see table 13C). These differences suggest that 
prior experience reduces some of the training benefits associated with 
the use of job aids; however, an analysis of individual tasks shows some 
Interesting differences that can still be attributed to differences in 
training. For example, approximately the same proportion of JPA and 
control enumerators with prior experience expected difficulties dealing 
with unusual situations on the end-of-trai ni ng questionnaire (37.7% and 
38. IX, respectively). However, after job experience the percent of JPA- 
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trained enumerators who thouqht chat this topic wa* not covered well 
emugh in training dropped by over 25 percentage points, whereu the 
control enumerators dropped by less than 3 percentage points. The drop 
in the proportion of JPA enumerators was statistically significant 
(x z « 64.2, 1 df, p * 0.00), whereas the change 1n the proportion of 
control enumerators was not (x 2 s 0.2, 1 df). 

One other comparison between enumerators with and without prior census 
experience was also of interest because it supported the assumption that, 
although the number of missing questionnaires was higher in the control 
group, the sample obtained was representative. For example, Table 14 
compares the percentages of enumerators who reported prior census 
experience. Although the enumerators in the groups (JPA and control) 
differed significantly in their levels of experience on both occasions 
(p< .05), the percentages of enumerators reporting prior experience changed 
little from one occasion to the next. 



TABLE 14 



Enumerators Reporting Prior Experience Working In The 1980 Census 





End-of-Traini ng 


Post-Trai n1"" 


JPA 


12.51 


11.5* 


Control 


17.3 


18.9 



As mentioned previously, one view of training is that it affects an 
enumerator's attitude toward both his/her job, as well as the census. 
In fact, it has been conjectured that poor training might affect an 
enumerator's decision to stay with a given job or to take a differe >, 
census job, once the present one was completed. Assuming that JPA 
training did a bettor job of preparing enumerators for their jobs, it 
was hypothesized that proportionally more JPA enumerators would express 
an interest to work in other (1980) census jobs. Responses of enumerators 
to the question, "If asked, would you work in another census job?" are 
presented in Table 15. Breakdowns by district office pairs are reported 
in Table 15A in Appendix F. 



TABLE 15 



Enumerators' Willingness to Work in Other Census Jobs 





JPA 


Control 


Yes 


61.0 


54.1 


No 


7.1 


11.7 


Depends on Job 


31.3 


34.2 
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Although JPA enumerators expressed significantly more unqualified Interest 
1n working in other census jobs ( x 2 * 11.7, 2 df, p ■ 0.00), 1t cannot 
be concluded that this difference was due solely to the JPA training. 
This finding must be interpreted cautiously in the context of other 
compa r1 sons. 

To determine what enumerators specifically disliked about their training, 
a space was provided on the end-of-train1ng questionnaire for comments. 
These comments were then collapsed by one coder Into major categories. 
Since respondents are usually reluctant to write responses on multiple-choice 
or checklist questionnaires, any category that was mentioned by at least 
one percent of the enumerators was reported in Table 16. 



TABLE 16 



Major Dislikes About Training Cited by Enumerators 



N = 1,168 




N = 1,104 




JPA 


% 


Control 


% 


Poor training facilities 


7 


Poor training facilities 


8 


Poor organization of training 




Poor organization of training 




(e.g., jumping around between 




(e.g., jumping around between 




manuals and workbooks) 


3 


manuals and workbooks) 


6 


Too long 


3 


Too long 


5 


Too much information presented 




Too short 


5 


1n two days 


3 


Too much information presented 




Too much paperwork 


3 


in two days 


5 


Too short 


3 


Too much repetit1on/tri via 


5 


Too much epetition/trivia 


3 


Sitting too long 


2 


Sitting too long 


2 


Lecture from training guide 


1 


Poor audiovisual s 


2 


Poor audiovisuals 


1 



P or training facilities led the 11st of complaints in both groups. Moreover, 
the lists of complaints were quite similar; the only differences were complaints 
about too much paperwork in the JPA training and lecture from the training 
guide 1n the control training. 



Supplemental Findings 

The following descriptive results were not dependent variables of interest 1n 
this e r>er1ment, but are presented because of their possible effects on 
enumenrlon during a census. 

An issue which has been of Increasing Importance, especially in urban enumeration 
districts where coverage may be affected, concerns an enumerator's willingness 
to work 1n neighborhoods where personal safety may be a real or Imagined worry. 
v able 17 presents the responses of enumerators to the question M Are there 
any neighborhoods Inyour assignment that you have been afraid to work in?" 
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TABLE 17 



Enumerators' Reported Fear of Working in Certain Neighborhoods 
in Their Assignment Areas 





Percentage Yes 




Cnnt Pol 


^ U RrnnHvn 


ifi 5 


Offices 


Pitts. E. 


19.2 




W. Columbus 


13.3 




Overall 


16.4 


JPA 


W. Queens 


16.3 


Offices 


Pitts. W. 


11.7 




S. Dayton 


14.4 




Overall 


14.7 



These results suggest that enumerators' fears about safety are not a 
minor problem. A chi-square comparison between the two groups for the 
overall responses was nonsignificant ( x 2 * .74, 1 df, p * .39) which 
Indicates that enumerator w fear M should not be a confounding variable 
1n this study. 



Enumerators who had expressed a fear of working in certain neighborhoods 
were then asked to check off or suggest alternative procedures that could 
be followed in those cases. Table 18 presents these major alternatives 
ranked in terms of their frequency of selection. The actual number of 
times an option was checked is presented in parentheses. 



TABLE 18 

Rankings of Suggested Procedures to Follow When an 
Enumerator is Afraid to Work in a Neighborhood 
in His/Her Assignment Area 



N * 952 N « 560 



JPA 




% 


Control 




% 


Work with another enumerator 


(109) 


80 


Work with another enumerator 


(60) 


65 


Enumerate only during daytime 


'67) 


49 


Enumerate only during daytime 


(37) 


40 


Work with crew leader 


(27) 


20 


Work with crew leader 


(23) 


25 


Send a different enumerator 


(14) 


10 


Enumerate alone 


(ID 


12 


Enumerate alone 


(13) 


9 


If people aren't home 




Leave those addresses to last 


(10) 


7 


first visit, don't return 


(11) 


12 


If people aien't home on first 






Send a different enumerator 




8 


visit, don't return 


0) 


7 


Leave those addresses to last 


6 


7 



NOTE: % is based on number of enumerators who answered "yes" to question about 
being afraid to work in certain neighborhoods in their assignment area. 



ERIC 



26. 



i^^rTT** team an< ? dayt1me ^mention led the list of suggests 
a^erT enumerat "* neighborhoods where personal safety "as 

On the post-training questionnaire, enumerators were also asked to m»* 

ihE Ch h a rr, that the Bureau could «k» to improve ?the enumerator's i^b 
Although tte instructions were to mark only thTee options, Tsub?tant af' 
number of enumerators marked all the options as If a 3-po nt sea e 

2! f ° + r i, th i S « uest1on - Accordingly, these responses were anaWzed bv 
ffi j 'Ml?^^ 8 J U 2 9eSted grower* was marked «^st * 

?J!r ' Rank1 f? s of suggested improvements are presented In Table 19 
for both groups. The number in parentheses represents the frequen?v 
that an improvement was marked "Most Important " irequency 



TABL 9 

Rankings of Suggested Improvements Given "Most Important" Rati 



ng 



N » 952 
JPA 

Pay more money 

Pay by the hour 

Pay on time 

Simplify the question- 
naire 

Allow overtime 

Allow team enumeration 

Allow work on Sunday 

Improve the training 

Simplify the job 

Use telephones for 
followup 

Better supervisors 





% 


432) 


46 


(343) 


37 


(267) 


29 


(216) 


23 


(160) 


17 


(133) 


14 


(102) 


11 


(89) 


10 


(70) 


8 


(61) 


7 


(26 


3 



N * 560 
Control 

Pay more money 
Pay by the hour 
Simplify the question- 
al re 
Pay on time- 
Allow overtime 
Improve the training 
Allow team enumeration 
Simplify the job 
Allow work on Sunday 
Use telephones for 

followup 
Better supervisors 





% 


(252) 


45 


(209) 


37 


(144) 


26 


(142) 


25 


(83) 


15 


(77) 


14 


(76) 


14 


(68) 


12 


(52) 


9 


(36) 


6 


(16) 


3 



H«f SJ^S: . b ! S !iV n rwo,ts frm wst "re"**, complaints about pay led the 

No further analyses of Items on either the end-of -training or Dost-traln,™ 

rarPwLrt- £n rePOrted AV h1S TOm0randum « The ng next me m r 

! lu 1 " J 111 P resent those results, as well as a discussion of 
Items on the Enumerator Performance Record. a.scussion or 
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Discussion 



Based on the results presented 1n this memorandum, 1t can be concluded that 
JPA training differed significantly from standard census training, although 
the differences were yenerally small as assessed by attltudlnal measures. 
Mo rever, both training approaches, at least 1n terms of enumerators giving 
positive ratings, appeared to be well received. 

The generally high ratings given both training approaches and the size 
of the differences obtained are not that surprising considering that 
most enume otors (except those with prior census experience) had no 
basis for comparison; that 1s, they did not receive JPA training followed 
by some standard training or vice versa. Further, since 1t can be assumed 
that enumerators do not come from a population with a great deal of 
recent exposure to formal training or educational settings, the rela- 
tively high ratings on all attltudlnal measures used are understandable since 
they were probably Influenced by the novelty of the situation (formal, struc- 
tured craving) and a "halo" effect; that 1s, a generally positive reaction to 
the training resulted 1n high ratings on all measures. These effects are common 
when attltudlnal measures are used. Nevertheless, 1t 1s Important to note that 
most of the differences found favored the JPA training, especially after job 
experience, and when prior experience working 1n the census was controlled 
for In the analyses. 

As with any field study, problems 1n Interpretation, particularly of the post- 
training results, remain. Serious field problems occurred 1n all the partici- 
pating district offices which, 1f more severe 1n the control group, could have 
depressed the attitude measures 1n those offices. On the other hand, there 
was anecdotal evidence that the higher return rates of evaluation data 1n the 
JPA group resulted from better support by field staff 1n those offices (District 
Office Managers and Field Operations Assistants) who saw significant differences 
in the alternative materials, appointed highly competent clerks, and supported 
those clerks 1n their effoits to obtain evaluation measures and administrative 
records used to develop enumerator performance records. Such Increased cooper- 
ation and support 1n the JPA offices, relative to the control group, could be 
expected to elicit cooperation rrom more marginal crew leaders, who would have 
been under rep resented 1n the control group. The effects such underrepresencatlon 
might have on the attitude and performance measures obtained are problematical, 
but 1t could be conjectured that both attitude and performance scores would be 
depressed 1f significant numbers of marginal crews were Included. On the other 
hand, 1t 1s also possible that the Increased support 1n the JPA offices caused 
more pos^Jve attitudes 1n both crew leaders and their enumerators, thereby 
raising their scores on the measures used 1n this study. 

Problems 1n Interpreting the data presented 1n this study, as well as specific 
recommendations for Bureau training programs, will be discussed fully in the 
next memorandum 1n this PERM series which will present the results of the 
field performance data. 
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TABLE 1 



Comparison of District Offices on Matching Variables 
Based on 1970 Census Statistics 





Pair 1 


Pair 2 


Pair 


3 




Pittsburgh 
West 


Pittsburgh 
East 


South 
Dayton 


West 

Columbus 


West 
Queens 


S. W. 
Brookl. 


Total 

Population 


626106 


62 2603 


704391 


689104 


644456 


651737 


% Black 
% Urban 


3.1 
88.3 


5.2 
98.2 


3.3 
68.4 


2.8 
80.9 


2.0 
inn 

IUU 


2.7 

IUU 


Median Age 


31.8 


31.7 


25.9 


26.7 


38.3 


33.0 


Median 
Education 


12.1 


12.2 


12.1 


* 


11.2 


11. A 


Median 
Income 


10290 


10606 


10942 


9856 


11168 


11040 


Allocation i/ Pop 
Rates Housing 


1.0 
2.2 


1.1 
2.1 


1.1 
1.8 


1.1 
2.1 


1.5 
2.4 


1.6 
2.5 


Mail -back 
rate 1970 


* 


* 


77.9 


78.0 







* - data not available 

i/ These rates are estimates of missing census data for population(Pop) and 
housing questions. 
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TABLE 2 

Percentage of Enumerators Trained on Selected Starting Dates in April 1980 



DATE 





April 


10 


11 


14 


15 


16 


17 


21 


22 


28 


Total 


Control 


S.W. Brooklyn 


0 


0 


88 


0 


0 


5 


0 


3 


1 


97 


Offices 


Pittsburgh E. 


21 


5 


41' 


5 


5 


2 


1 


6 


7 


93 




W. Columbus 


0 


0 


0 


22 


25 


0 


43 


10 


1 


101 


JPA 


W. Queens 


0 


0 


8 


83 


6 


0 


3 


0 


0 


100 


Offices 


Pittsburgh W. 


0 


0 


95 


0 


0 


0 


0 


0 


1 


96 




S. Dayton 


0 


0 


97 


0 


0 


0 


0 


0 


0 


97 



Note: An excluded date accounts for, at most, AX of the total cases for a 
district office. Rounding resulted in the total of 101% for line 3. 

, , , « 
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Illustration 7.A 
DETERMINING THE TYPE OF QUESTIONNAIRE 

ADDRESS LABELS 
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7.B DETERMINING THE STATUS OF THE HOUSING UNIT 

0 

The procedures you follow during enumeration will vary, based on whether you determine 
the present status of the housing unit to be occupied, vacant, or nonexistent. 

7.B1 Occupied Housing Units 

a Occupants Temporarily Absent 

If parsons usually reside at the housing unit, but they are temporarily 
absent, consider Che housing unit occupied* Determine (from a neighbor, 
building manager, and so forth) when the occupants will return* Enter 
this information in column (13), Remarks, of the Adaress Listing Page 
on the line for the housing unit and return at that time* 

• 

If the occupants are not expected to return before the end of followup I, 
obtain all the information you can* You must get at least ''last resort 11 
information (see chapter 9, section 9.E)* 
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b Housing Unit Occupied Entirely by Persons Who Did Hoc Live There on Census 
Day 

If the respondent volunteers that ell of the current residents did not live 
in the unit on Census Day (April 1, 1980), and: 

(1) If Che respondent says that he/she completed a questionnaire and 
•ailed it back to a census office from another address, then 
enumerate the unit as either: 

(a) VACANT, if no one lived there on Census Day; or 

(b) OCCUPIED, if someone else lived there on Census Day; obtain at 
least "last resort 19 ir formation about the former residents, as 
instructed in chapter 9, section 9.E. 

(2) If the persons now living in the housing unit were not enumerated at 
any address, then enumerate them at the address. In this case, do 
NOT obtain information about former residents. 

c Housing Unit Occupied Entirely by Persons with a Usual Hot ~ Elsewhere (UHE) 

If a respondent says that all persons residing in the housing unit have a 
usual home elsewhere, enumerate the housing unit as "Vacant, UHE" (see 
section 7.F). 

• , 

d Living (garters Occupied by Nine or More Persons Unrelated to the Person 
Who Owns or Rents the Housing Unit 

If a living quarters is' occupied by nine or more persons unrelated to the 
person(s) who own(s) or rents the housing unit or by ten or more unrelated 
persons, it is a "speci* * place*" Do not complete questionnaires for special 
places (see chapter 9, section 9.K). 

7.B2 Vacant Housing Unit 

A "For Sale" or "For Rent" sign is NOT proof that a housing unit is vacant. 
If no one answers the door, contact a neighbor, janitor, rental agent, real 
estate agent, or building manager to determine whether the housing unit is 
actually vacant. 

Enumerate the housing unit as vacant if no one is living there at the time of 
your visit AND one or more of the following conditions exist: 

a It is awaiting occupancy. 

b It is under construction and the final windows and doors are in place. 

c It is a vacation cottage or similar place used only on weekends or during 
certain times of the year. 

d It is used for storage of household furniture. 
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U.S. Ot^ARTMCNT Of COMMCHC 
• UMCAu OF TWC CCNJU 



ENUMERATOR TRAINING EVALUATION 

20th Decennial Census - 1980 



— 

'* m INSTRUCTIONS: The Bureau of the Census 
wants to know how you feel about the 
training that you just finished* 

Please answer the questions below. Do not 
write your name on this form. 



ENTER YOUR 4-O/C/T DISTRICT OFFICE CODE NUMBER HERE 



1. Is this your first job working in tho 1980 census? 
i □ Yts 
No 



2. How woll prepored do you fool to 90 out and work, 
os on enumerator? 

Mark (X) one 

1 Q Well prepared 

2 Q Adequately prepared 

3 □ Poorly prepared 

4 Q Not prepared 



3. Whero wore you trained? 

1 I I In a room with 3 or more people 

2 □ On the job — How long did tho training lost? 



. Hours 



4. Tho following is o list of difftront activities used 
in training* 

Mark an (X) in the appropriate column that best 
• describes how eocn activity helped you /earn 
your job. 



a* Lecture by trainer 
(reading from training 
guide) 

b. Practice interviewing 
(either with trainer or 
fellow enumerators). . . . 

c. Workbook exercises . . . . 

d. Reading or using 
manuals 

o« Audiovisuals (slides and 
tape recordings) 

f. Other - Specify^ 



Very 
helpful 


Helpful 


Least 

helpful 


1 


2 


3 


t 


Z 


3 


t 


z 


3 


t 


2 


3 



5* Wo* tho length of training - 
Mark (X) one 
« Q Too short? 

2 □ About right? 

3 □ Too long? 



6. Overall, how would you rato tho quality of the 
training? 

Mark (X) one 
1 □ Very good 
z □ Good 
3 □ Fair 
Poor 



7* Woro the training materials easy to read and 
understand? 

ID Yes 

2D No 



8. The following is a list of different parts of your 
job. 

Alark an (X) by each that you think will cause 
some problems when you start working. 

1 Q Locating addresses that did not return 
a questionnaire 

z Q Interviewing people (asking the questions) 

3 □ Completing the questionnaire (address label, 
"For Census Use Only" box, etc.) 

♦ □ Completing the address register 

5 Q Filling out payroll forms 

s □ Using the job reference manuals 

7 Q] Getting people to cooperate 

s Q Dealing with unusual situations 

9 Q Answering questions asked by respondent 

10 Q Checking for missed housing units 

1 1 Q Other - Specify^ 



9. What ono thing did you dislike most about the 
training? 
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U.S. OCPAHTMtNT OF COMMCftCC 
•UA^AU 0^ TM| CENSUS 



TRAINING ANO PERFORMANCE EVALUATION 
FOLLOWUP ENUMERATOR 

20th Decennial Census - 1980 



INSTRUCTIONS: Before yea began working, you com- 
pleted a questionnaire about your training for mis job. 
Now that you have had some job experience, the Bureau 
of the Census wants to know how that training helped 
you in your job, and how you feel about your job. 

Please answer the questions below. Do not write your 
n ie on this form. 



ENTER YOUR DISTRICT OFFICE CODE NUMBER HERE 



U la this your first job working in the 1980 census? 
i Q Yes 
a □ No 



2. How well prepared were you to go out and work as 
on enumerator? 

Mark (X) one 

t Q Well prepared 
a □ Adequately prepared 
a Q Poorly prepared 
* Q Not prepared 



3* Where* were you trained? 

t | 1 In a room with 3 or more people 
z □ On the job 



4. The following is o list of different ports of your 
job. 

Mark on (X) by each mat was or covered 
we// enougn In training. 

i rZ Locating addresses that did not return 
a questionnaire 

t Interviewing people (asking the questions) 

3 □ Completing the questionnaire (address label, 
"For Census Use Only" box, and item 2 on 
back page) 

4. | ! Completing the address register 
(columns 10 and 13) 

s Q] c lUing out payroll forms 

> I j Using the job reference manuals 

7 [_j Gluing people to cooperate 

• □ Dealing with unusual situations 

* □ Answering questions asked by respondents 
i o Checking tor mused homing units 

1 1 □ Other — Specify 



5» The following is o list of different activities 
used in training. 

Mark on (X) in the appropriate co/umn that best 
describes how each activity helped you team 
your job, 



a. Lecture by trainer 
(reading from Gaining 
guide) 

b. Practice interviewing 
(either with trainer or 
fellow enumerators) . . . . 

c» WorkbooK exercises 

d. Reading and uimg 
manuals 

e. Audsovirials (slides and 
tape recordings) 

f. Other - Specify. 



Most 
helpful 



Helpful 



Least 

helpful 



6* How would you rote your own job performance 
so far? 

Mark (X) one 

1 □ Vef y * ood 

z □ Good 

3 C3 ^ a,r 

4. □ Poor 



7. If asked, would you work in another census job? 
.Mark (X) one 

t G Yes 

* QNo 

a Q Depends on the job 



erJc 



I. On the evereee. Hew (Mny times eaeii day do 



o. I»wNfit«f mental er job eid maool? 



b. Questionnaire refofeiiee book? 



9. Overall, Ww w««U yee rot* the quality of the 
training you. received for this job? 
Mark (X) 



i □ Very toed 
iQ Good 
iQFiir 
4 Q Poor 



10* T«t# following is * list of suejested imorevootents. 
Mark (X) m order of importance the three changes 
which you think the Bureau of the Census cou/d 
make to improve your job 



a. Improve the training . • 

b. Fay more money 



c* Simplify the job (get rid 
of the manuals) 



d. Allow work on Sunday 



e* Atl^w overtime work • . » 

f. Pay by the hour* not 
the questionnaire 

e. 0o all interviewing using 
the telephone 

h. Have enumerators work 
in teams . . . . 

I. Simplify the 

questionnaire 

|» Have better 

supervisors . 

k. Pay enumerators on 
tine . • • 



L Other - Specify 



11. Are there any neighborhoods in your assignment 
thet you hove been efreid to work in? 

i Q No 

Q Yes - What should the Bureau let you do 
about it? Mark all that apply 

z O nothing special, just do your *ormal job 

3 Q Leave those addresses to last 

4 Q Let you work with another enumerator 

s □ Ask another enumerator to to there 

s Q Not to back if people are not home the 
first time 

r Q Tell your crew leader :o work with you 
s □ Go there only during the daytime 
» □ Other - Specify. 



c 

5 

I* 


a*- 

c 
m 

•*» 
to 


jSlf 




2 


* 




2 


3 • 


■ 


Z 


3 




2 


3- 




Z 


3 




2 


3 




2 


3 




2 


3 




2 


3 




2 


3 




2 


3 




2 


3 



12. What wos your employment status when you were 
• hired for the 1980 census? 
Mark (X) one 

t □ Looking for a full or part-time job 
(temporary or permanent) 

2 r2 Not working, not retired, not a student. 

not looking for a job 

3 □ Retired 

4 □ Student 

3 Q Employed, job starts in the future or 
temporarily laid off 

• □ Employed 

7 ^ Something else - Specify, 
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Form D 275 - Record of Train ng 



»U«CAU O* TH« ClNlu» 



ALTERNATIVE TRAINING EXPERIMENT 

FOLLOWUP I ENUMERATOR 
PERFORMANCE RECORD 

20th Decennial Ctfisvs - 1980 



1. 0.0. code 


2. C.L.D. No. 


3. Training dates 
April __to April 


4. Enumerator name 


5. Telephone No. 


6. ED No. (0000, if reserve) 


7. Review test score 


*• Did the enumerator complete training? Mark (X) one 
i Q y es z ^2 No 3 □ Don't know 



Forms 0*267 A or 0*2678 - Field Employ** 
Selection Aid (Enter • - ?9" if no test score) 



Form 0*170 — QC Enumerator Daily Progress Record 

(Use first two forms for each crew leader district.) 



16. Date (item 6) 
April 


17. No. of acceptable questionnaires 


18* Date (rem 6) 
April 


19. No. of acceptable questionnaires 



9. Selection test score (item 0) 



Form 0-185 - Record of First Review 

(Enter "99" if form is not available.) 



10. Number of M NoV \» Part A 



11. Part A results 
Mark (X) one 

t ; | Sausfactory 

2 ^2 Needs improvement 

3. Unsatisfactory 

♦ t No form completed 



12. Number of short form erro.'S 



13. Number of long form errors 



14. Part 8 results 
Mark (X) one 

iQO-4 

z □ S-20 

3 O More than 20 

4 n No form completed 



Form 0*152 — Crow Loodor Record of Assignments 

(Use at end of FoIIqwuqL) 



20. Did enumerator complete ED? {"X 1 * in front of ED No.) 
Mark (X) or.e 



1 □ Yes 

2 ~ No 

3 ~ Don": know 

Not on form 



Form 0*291 — Employee Pay Vouchor 

(Check at end of Fo//owupL) 



21. P?3Son for separation 
Mark (X) one 

1 I Assignment completed 

2 Cj ^ av dissatisfaction 

3 Q Work dissatisfaction 

4 Q To take another job 
s Q Poor performance 

* No form ava. fable 
7 □ Other ~ Specify ^ 



Form 0-158 — Record of Reinterview 

(Enter 4# 99** if person's name is not on form.) 



Form 0-153 - Crow Loodor Record of Progress 

(Use at end of Fo//owup I) (Enter "999" if person 
is not on form.) 



15. Number of fails (Look for more chart 
one box if there ore any fat Is.) 



22. Copy che third entry for each enumerator 
or last entry) 
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TABLE 3B 

District Office Comparisons For Degree Of Preparedness 
Question: "How well prepared do you feel to go out and work as an enumerator?" 

End-of-Tral n1ng 

(391) (290) (312) (286) (455) (509) 

*W. Queens S.W. Brook. *P1tts. W. Pitts. E. *S. Dayton W. Col. 

Well Prepared 52.4% 59.7% 46.5% 50.7% 44.81 40.3% 

Adequately Prepared 46.8 38.6 52.6 47.9 54.9 58.0 

Poorly Prepared 0.8 1.7 1.0 1.4 0.2 1.6 

Not Prepared 0.0 0.0 0.0 0.0 0.0 0.2 



Post-Training - 

(393) (123) (158) (217) (368) (217) 

*W. Queens S.W. Brook. *P1tts. W. Pitts. E. *S. Dayton W. Col. 

Well Prepared 65.1% 61.0% 63.3% 58.1% 55.2% 47.9% 

Adequately Prepared 34.1 34.1 36.1 39.6 42.9 50.7 

Poorly Prepared 0.3 4.1 0.6 2.3 1.9 1.4 

Not Prepared 0.5 0.8 0.0 0.0 0.0 0.0 

*JPA offices; base of precentage 1s in parentheses above district office name. 
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TABLE 3C 

Chl-Square Statistics for District Office Comparisons 



W. Queens A Pitts. W. & S. Dayton & 

S.W. Brook. Pitts. E. W. Col. 

2 2 2 

X p X p X p 

End-of-T raining 5.4 .07 1.4 .49 7.2 .07 

Post-Training 12.1 .01 2.3 .31 3.4 .19 
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TABLE 3D 



Comparison of Changes In Responses (from end-of -training to post-training) 
On the Preparedness Question for Each District Office 



District Office 


2 

X 


df 


P 


W. Queens 


0.0 


1 


1.00 


Pittsburgh W. 


0.0 


1 


1.00 


S. Dayton 


4.4 


1 


0.04 


S.W. Brooklyn 


2.2 


1 


0.14 


Pittsburgh East 


0.2 


1 


0.68 


W. Columbus 


0.0 


1 


0.96 



Note: "Well" and "Adequately Prepared" responses were collapsed 
into a prepared category, and "Poorly" and "Not Prepared" 
responses were collapsed into a not prepared category for 
the above analysis. 



TABLE 7A 

District Office Comparisons For Quality Of Training 
Question: "Overall, how would you rate the quality of the training?" 



End-of-T raining 



Very Good 
Good 
Fair 
Poor 



(390) 
Queens 


(294) 
S.W. Brook 


(315) 
*P1ttS. W. 


(285) 
Pitts. E. 


(458) 
*S. Dayton 


(514) 
W. Col. 


52. 8% 


55.4% 


57.lt 


52. 6% 


60.5% 


43.4% 


41.0 


34.4 


36.5 


38.9 


36.0 


48.2 


5.6 


9.9 


5.1 


7.7 


3.1 


7.8 


0.5 


0.3 


1.3 


0.7 


0.4 


0.6 



Post-Training 

(380) (114) (153) (212) (354) (200) 

*W. Queens S.W. Brook *P1tts. W. Pitts. E. *S. Dayton W. Col. 

Very Good 52.6% 49.1% 70.6% 45.8% 60.2% 41.5% 

Good 43.2 40.4 26.1 44.8 35.3 52.0 

Fa1r 2.9 7.9 2.6 9.4 3.4 6.5 

Poor L3 2.6 0.7 0.0 1.1 o.O 



*JPA offices; base of percentage Is in parentheses above district office name. 
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TABLE 7B 

Chl-Square Statistics for District Office Comparisons 



W. Queens & Pitts. W. & S. Dayton & 

S.W. Brook Pl.ts. E. W. Col. 

x 2 x 2 v 2 

P A p X p 

End-of-Tralnlng 6.3 .10 2.9 .40 32.1 .00 



Post-Training 6.7 .08 25.8 .00 21.9 .00 



TABLE 7C 

Comparison of Changes in Responses (from end-of -training to post-training) 
On the Quality Question for Each Oi strict Office 



District Office 


2 

X 


df 


P 


W. Queens 


1.1 


1 


0.29 


Pittsburgh W. 


1.4 


1 


0.24 


S. Dayton 


0.3 


1 


0.57 


S.W. Brooklyn 


0.0 


1 


1 .CO 


Pittsburgh E. 


0.1 


1 


0.8i! 


W. Columbus 


0.5 


1 


0.50 



Note: "Very Good" and "Good" responses were collapsed into an acceptable 
category, and "Fair" and "Poor" responses were collapsed into an 
unacceptable category for this analysis. 




TABLE 13A 

Percentage of Enumerators Checking Job Tasks Prior to And After Job Experience 

End-of-Training question: M The following is a list of different parts of your job. Mark an (X) by each that you 

think will cause some problems when you start working." 

Post-Training Question: "The following is a list of different parts of your job, Mark an (X) by each that was 

not covered well enough in training," 



End-of-Training 



Prior Experience 



No Prior Experience 





(146) 

IDA 

J PA 

a. 


(189) 
Control 
b. 


(335) 
Overal 1 
c. 


(1021) 
JPA 
d. 


(905) 
Control 
e. 


(1926) 
Overal 1 
f. 


Locating addresses 


7.5% 


8.5X 


8.1% 


8. OX 


6.6% 


7.4% 


Interviewing people 


16.4 


11.1 


13.4 


11.0 


8.7 


9.9 


completing tne questionnaire 


a o 
o.o 


4.2 


5.4 


7.8 


5.9 


6.9 


Completing the address 
register 


2.7 


4.2 


3.6 


4 9 


6 4 


J. u 


Filling out payroll forms 


4.8 


12.7 


9.3 


8.7 


15.5 


11.9 


Using job reference manuals 


7.5 


13.2 


10.7 


4.8 


8.4 


6.5 


Getting people to cooperate 


45.2 


45.5 


45.4 


50.0 


35.0 


43.0 


Dealing with unusual 
situations 


37.7 


38.1 


37.9 


43.4 


39.3 


41.5 


Answering respondent 
questions 


8.2 


10.1 


9.3 


10.8 


10.9 


10.9 


Checking for missed housing 
units 


11.6 


15.9 


14.0 


20.6 


19.2 


19.9 



NOTE: Base for percentages 1s reported In parentheses above each column. 



TABLE 13A (Continued) 



Post-Tralnlna 

Prior Experience 





(107) 
JPA 
a. 


(106) 
Control 
b. 


(213) 
Overall 
c. 


Locating addresses 


5.6% 


5.7* 


5.6X 


Interviewing people 


2.8 


0.9 


1.9 


Completing the questionnaire 


2.8 


5.7 


4.2 


Completing the address 
register 


3.7 


6.6 


5.2 


Filling out payroll forms 


1.9 


10.4 


6.1 


uailiy JUU 1 CI CI CMtC IHullUCI 1 D 


1 Q 


7 a 


A 7 


Getting people to cooperate 


15.0 


17.9 


16.4 


Dealing with unusual 
situations 


12.1 


35.8 


23.9 


Answering respondent 
questions 


2.8 


5.7 


4.2 


Checking for missed housing 


1.9 


14.2 


8.0 



units 



NOTE: Base for percentages Is reported in parentheses above each column. 



No Prior Experience 
(820) (454) * (1274) 



JPA Control Overall 

d. e. f. 

8.7X 10. 4X 9.3X 

7.4 8.1 7.7 

5.5 9.7 7.0 
4.8 9.7 6.5 

12.6 17.0 14.1 
4.8 8.8 6.2 

14.8 21.4 17.1 

18.7 26.2 21.4 

7.4 9.0 8.0 

14.1 15.9 14.8 
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TABLE 13B 



Chl-Square Statistics For JPA and Control Groups Controlling for Prior Experience 

End-of -Training 

» 

JPA Control Overall 





2 

X 


P 


2 

X 


P 


X 


P 


Locatlno addresses 


0.0 


0.96 


0 6 


0 46 


fl 1 


fi 1A 
U. / *t 


Interviewing people 


3.2 


0.07 


0.8 


0.37 


3.4 


0.06 


Completing the questionnaire 


0.0 


0.8O 


0.5 


0.48 


0.8 


0.36 


ComoletinQ the address 

w w "f lowing v*iw u v* ui «• «• 

register 


0.9 


0.34 


0 9 




1 9 


n i 


Filling out payroll forms 


2.1 


0.15 


0.7 


0.40 


1.7 


0.19 


Using job reference manuals 


1.4 


0.23 


3.8 


0.05 


7.2 


0.01 


Getting people to cooperate 


1.0 


0.31 


6.9 


0.01* 


0.6 


0.45 


Dealing with unusual 
situations 


1.5 


0.22 


0.1 


0.81 


1.4 


0.24 


Answering respondent 
questions 


0.6 


0.42 


0.1 


0.82 


0.6 


0.44 


Checking for missed 
housing units 


5.9 


0.01* 


0.9 


0.33 


6.1 


o.or 



♦Indicates significant difference, p < .05 



JPA 



2 

X 



Locating addresses 0.8 

Interviewing people 2.5 

Completing the questionnaire 0.9 

Completing the address 0.1 
register 

Filling out payroll forms 9.7 

Using job reference 1.2 
manuals 

Getting people to cooperate 0.0 

Dealing with unusual 2.3 
situations 

Answering respondent 2.5 
questions 

Checking for missed 11.8 
housing units 



♦Indicates significant difference, p < .05 
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TABLE 136 (CONTINUED) 
Post-Training 

Control Overall 

P 2 p 2 * p 

X X 

0.37 1.7 0.19 2.6 0.11 

0.12 6.0 0.01* 8.8 0.00* 

0.34 1.3 0.26 1.8 0.18 

0.82 0.7 0.42 0.4 0.55 

0.00* 2.3 0.13 9.7 0.00* 

0.26 0.1 0.82 0.5 0.48 

1.00 0.4 0.51 0.0 0.88 

0.13 3.5 0.06 0.6 0.45 

0.12 0.9 0.35 3.2 0.07 

0.00* 0.1 0.77 6.5 0.01* 



TABLE 13C 

Chl-Square Statistics for JPA vs Control Groups Controlling for Prior Census Experience 



End-of-fralnlnq 



Post-Training 





Prior Experience 
2 

X p 


No Prior Experience 
2 

X p 


Prior Experience 
2 

X p 


No Prior Experience 
x 2 


Locating addresses 


0.1 


0.91 


1 2 


0 ?fi 


n n 


i nn 

km uu 


n q 


U. SI 


Interviewing people 


1.6 


0.21 


2.5 


0.12 


0.2 


0.60 


0.1 


0.73 


Completing the 
questionnaire 


0.7 


0.42 


2.6 


0.11 


0.5 


0.49 


7.3 


0.01* 


completing the address 
register 


0.2 


0.67 


1.8 


0.18 


0.4 


0.53 


10.9 


0.00* 


Filling out payroll 
forms 


5.2 


0.02* 


20.2 


0.00* 


b.3 


0.02* 


4.3 


0.04* 


Using job reference 
manuals 


2.2 


0.14 


9.7 


0.00* 


2.7 


0.10 


7.6 


0.01* 


Getting people to 
cooperate 


0.0 


1.00 


43.6 


0.00* 


0.2 


0.69 

• 


8.5 


0.00* 


Dealing with unusual 
situations 


0.0 


1.00 


3.1 


0.08 


15.1 


0.00* 


9.5 


0.00* 


Answering respondent 
questions 


0.1 


0. 70 


0.0 


0.97 


0.5 


0.49 


0.8 


0.37 


Checking for missed 
housing units 


0.9 


0.34 


C.5 


0.50 


9.3 


0.00 


0.6 


0.46 



♦Indicates significant difference, p <. .05. 
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TABLE 15A 



Yes 
No 

Depends on 
the job 



District Office Comparisons of Desire for Future Census Employment 
Question: "If asked, would you work 1n another census job?" 



(397) 
*W. Queens 

62. OS 

7.3 

30.7 



(121) 
S.W. Brook. 

51.2% 

15.7 

33.1 



(158) 
*P1tts. W. 

63. 9% 

8.9 

27.2 



(216) 
Pitts. E. 

51.4% 

9.3 

39.4 



*JPA offices; base of percentage 1s 1n parentheses above district office name. 



TABLE 15B 

Ch1-Square Statistics for Office Pairs 



(369) 


(218) 


Dayton 


W. Col. 


58.8* 


5^.3X 


6.2 


11.9 


35.0 


29.8 



M. Queens vs S.W. Brook. 
Pitts. W. vs PUts. E. 
S. Dayton vs W. Col. 



9.0 
6.5 
6.4 



0.01 
0.04 
0.04 
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Report Number 2 



SUMMARY 



Overview 



lj «} °IL V?/'?^ 1n ths s er i es (Pr , No< 9) descr1bed the desi ^ } 
T^LliOZ Alternative Training Experiment and compared reactions of followup 1 
f ?5^22« ♦! Va ^ y ° att l tud1nal ^ures. The purpose of this memorandum 
n«K?ofy the performance of enumerators trained using alternative materials 
iSSSlTrSSSn * train l ng ] w1th * he Performance of enumerators trained using 
!2^J^rl5 • V C ? n ! US " Iater1als - Th e Performance measures analyzed included 
^:?I^ r f! n L n L kn 2 w1ed9e , t !! t ?! self -reP°rt measures, and quality Control records 



obtained as part of normal field operations. 

fl^ e « th ! 1nt ?"Ji 0f PERM r ??? rts 1s t0 C1 >culate infomatlon among Bureau staff 
members as quickly as possible, It 1s emphasized that this memorandum is a 

5£lr! iniZ TSSt' / S SUC -!l' add1t1onal anal y"s are Hkely to be done and, 
accordingly, conclusions and recommendations may change. 

Summary of Results 

JortXfSffEJJ N0< 9 ; d ! f k ferent1al «tes of missing data in the JPA and 

SE^J»2 I ^\ alS ° ? ccurred w1th mos t of the performance measures described in 
•InlrXX'n. a Ij, a measures of enumerator production were below pre-census 

According y, It was concluded that field problems, primarily problems 
with address registers, lowered production levels and confounded experimental com- 

ltZllT 0 f:l n9 ?5 e PePl0d !" f0ll0Wup 1 of most 1nterest for assessing the 
effects of the alternative training approach. 

!Ji h ?L f1n ii n9 f, als0 ra1sed ser1ous ^estlons about the reliability and validity 
»«™ ion measures. For example, 1 ntercorrel atlons among production 

3!«rJ 52* ?? Vh 3 5 t0 m0derate relationships, even though the measures 
were similar and collected 1n contiguous time periods. Further, the production 
measures, which were collected during the Initial days of followup 1, were of no 
value in predicting the closing dates of the particijati ng district offices a 
result that brings Into question the validity of the measures as true indicators 
V^JX°V*il °l f I eldwork in a district office. These findings, buttressed by 
ancedotal ,eedback about the prevalence of field problems, precluded any definitive 
statements about the effects of the alternative training on job perfonnlnce. 

nll^iit n ° conc J us :° ns ab ° ut the effec ts of training on job performance were 
Possible, several other findings were of interest and raised important issues in 
the planning of future censuses and conduct of experimentation. Specifically: 

Extreme variation between district offices was found on many of the 
measures analyzed. Accordingly, the utility of conducting experimental 
comparisons with small sample sizes of district offices was questioned. 



1. 



2. Recordings of quality control measures used in this study wer» flawed for 
two reasons 1 



a. in the face of pressure to maintain production and keep 
the census on schedule, they were either not done or, 

b. if done, procedures were often not followed correctly. 
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3. Paying enumerators an hourly wage at some point during followup 1 
significantly reduced attrition. Moreover, rates of enumerator 
attrition were significantly related to the closing dates of district 
offices, suggesting that paying hourly w; ies may be a more cost-effective 
strategy than paying piece-rate wages 1n future censuses. 

4. Relatively low rates of manual use by inexperienced followup 1 enumerators 
questioned the assumption on which much of decennial training was based. 
Specifically, certain topics were only briefly covered in training because 
It was assumed that enumerators would refer to their manuals if the need 
arose. 

5. Verbatim training apparently achieved the goal of delivering standardized 
training, I.e., the average scores of enumerators on training review tests 
did not differ significantly in c ; 1strict offices receiving either the 
alternative (JPA) or standard (control) training. 

Outline of Paper 

[? a ! s , 1 ! t the reader 1n identifying specific areas of intent in this memorandum, 
toe following topical outline 1s presented.' 



Topic 

I. PURPOSE 

II. PERFORMANCE MEASURES 
III. RESULTS AND DISCUSSION 

a. Final P«*v1e* Tests 

b. Self-Report of Enumerator Performance 

c. Self -Reported Average Dally Use of Manuals 

d. Cumulative Number of Questionnaires Completed 

t. An Enumerator After Three Days of Work 
1n Followup 1 

e. Qt*»11ty Control Enumerator Progress Report 

f. Record of First Review (D-185) 

g. Record of Relntervlew, Form D-158 

h. Other Variables of Interest 

1. Method of Payment and Enumerator Attrition 
j. Intercorrelatlons Among Performance Varlajles 

IV. RECOMMENDATIONS 

V. APPENDIX 

List of Tables and Figures 

Sources of Performance Oat* and the Percentage of 
Enumerators 1n the JPA and Control Grojps with 
Usable Data 

Enumerators' Self-Reported Job Performance 
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September 10, 1981 




UNITED STATES DEPARTMENT OF COMMERCE 
Bureau of the Census 

Washington. O.C. 20233 



1980 Census 

PRELIMINARY EVALUATION RESULTS MEMORANDUM NO. 12 

Prepared by: William Mockovak, Center for Social Science Research 

Subject: The Alternative Training Experiment: Analysis of 
Performance Data, Discussion, and Recommendations 

Purpose 

A previous n^wrandum 1n this series (PERM No. 9) described the rationale for 
and development of alternative enumerator 'training materials that stressed the 
use of job performance aids (JPA's)i/ 1n both the followup 1 enumerator's manual 
and classroom training. That memorandum also compared the reactions of 
enumerators, on a variety of attitudlnal measures, who had been trained using 
either job performance aids (JPA group) or standard census reference materials 
(control group). The purpose of this memorandum 1s to present additional com- 
parisons on a variety of performance measures that were collected during this 
experiment, which was one of six experimental programs conducted as part of the 
1980 Decennial Census Evaluation and Research program. In addition, limitations 
of the data (both attltudinal and performance) and recommendations for Bureau 
training programs will also be discussed. 



1/ Job performance aids 1n the context of this experiment refer to alternative 
formats (e.g., flowcharts, checklists, decision tables) for presenting pro- 
cedural information that would typically be presented in paragraph form. 



Note: The data in this report are preliminary and tentative in nature. Users 
of the results memoranda should understand that these documents are prepared for 
Internal office use, with the aim of circulating information among Bureau staff 
members as quickly as possible. These memoranda, therefore, do not undergo the 
careful review and clearance normal ly associated with published census documents. 
Conclusions and recommendations contained herein essentially reflect the thoughts 
of certain staff members at a point in time, and should not be interpreted as 
statements of Bureau position, 
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Performance Measures 



2. 



Rather than attempt to develop a single, overall measure of enumerator job 
performance, this study used a variety of Individual Indices Including self- 
report n^asures, test scores after training, Quality Control enumerator 
reports, and crew leader quality control and administrative records. Following 
in Table 1 1s a listing of each measure used and the percentage of enumerators 
from the JPA and control groups with data available for analysis. 



TABLE 1 

Sources of Performance Data and the Percentage of Enumerators 
1n the JPA and Control Groups with Usable Data 



Measure 

!. Self-report of enumerate - performance 

2. Self- report of manual 

(JPA or standard) use 

3. Self-report of Questionnaire 

Reference Book (QRB) use 

4. Final review test score after training 

5. Cumulative number of questionnaires 

reportedly completed by enumerator 
after 3 days of work during 
followup 1 

6. Number of acceptable questionnaires 

completed by an enumerator as re- 
ported on the f i rst QC Enumerator 
progress record 

7. Number of acceptable questionnaires 

completed by an enumerator as 
reported on the second QC Enumerator 
progress record 

8. Results of a crew leader's "First Re- 

view" of an enumerator's work 

9. Results of n crew leader's verlfica- 

t* n of household cor.^osition as 
reported by followup 1 enumerators 



Sou rce 

Post-training questionnaire 

Post-training questionnaire 

Post-training questiornai i 
Classroom training 



D-153, Crew Leader 
Record of Progress 

D-170, QC Enumerator 
Daily Progress Record 



D-170, QC Enumerator 
Daily Progress Record 

D-185, Record of 
First Review 

D-15c, Record of 
Interview 



N-1J77 

JPA 

76.6% 

76.4 

76.4 
85.4 

90.1 



84.7* 



N-1,349 

Control 

AO. 7% 

39.4 

39.4 
79.1 

85.0 

65.0* 



77.7* 51.5* 
80.0* 70.0* 
36,4* 29.4* 



* These are conservative estimates because the denominators include enumerators 
who quit or were released prior to the completion of the fom(s). 
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As noteo 1n the previous memorandum In this series (PERM NO. 9), the JPA 
group had higher return rates (fewer missing data) for each of the measures 
used. However, as Table A 1n the appendix shows, this difference was not con- 
sistent for each district office pair across all evaluation measures. 

Two problems with the evaluation measures became apparent early 1n the study, 
one of which 1s reflected 1n the figures shown 1n Table 1. First, the evaluation 
measures were not being completed and, second, when completed, the established 
procedures for completing the measures were often ignored. For example, the 
data-capture clerk in the Pittsburgh East district office reported that despite 
personally calling all the crew leaders, several still did not complete the 
forms required for first review (D-185) or the telephone reinterview (D-158). 
Moreover, the Senior Office Cleric (SOC) for the Field Operations Supervisor (FOS) 
failed to collect or file any of the quality control data 1n the district office. 
Such lack of support from office and field personnel obviously affected the 
return rates of the evaluation measures for this district office. 

The second problem, procedures not being followed when forms were completed, is 
based primarily on anecdotal reports from data-collection clerks and reviews of 
samples of forms. These apparent shortcomings will be discussed more fully 
when each evaluation measure 1s discussed in turn 1n the n^ *t section of this 
report. One obvious example, however was the completion of a form by Field 
Operations Assistants (FOA's) which ' ,1mated the length of training for follow- 
up 1 enumerators. Although both crew leaders and FOA's were reassured that this 
form would not affect an enumerator's pay for training (enumerators were to be 
paid for 2 days of training), most estimates of training time were exactly 16 
hours, the amount of time prescribed by the crew leader manual* When crew leaders 
1n one office were asked about this unlikely consistency, they reported that they 
did not trust reassurances about pay not being reduced 1f training finished before 
the allotted time had expired. 
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Results and Discussion 



Before finishing training, enumerators 1n both the JPA and control groups were 
required to complete final tests that reviewed the major concepts and content 
of the training. Reflecting the differing goals and learning activities of 
the writers who prepared them, these tests varied both 1n content and purpose. 
For example, the JPA test was designed as a final criterion measure; that is, 
enumerators either got a minimum number of test items correct or else remained 
1n class for additional classroom instruction. This requirement to obtain a 
minimum passing score on the final review test was not a feature of the control 
training. Instead, it was designed merely as a means for reviewing major job 
concepts. 

Since comparisons between training methods were not possible, a one-way analysis 
of variance (ANOVA) was computed separately for each group. The purpose of this 
analysis was to determine if district offices in either group differed on their 
mean test scores. Such differences, 1f found, could reflect either a failure 
of a training approach to deliver standardized training or some other possible 
systematic difference between offices, such as differences 1n the education of 
enumerators or in the quality of trainers. 

To ensure standardization of scoring, a sample of 84 tests was randomly selected 
from each district office and scored by the same person. As the results in 
Table B of the appendix show, the hypothesis of equality of mean test scores 
between JPA offices could not be rejected. However, the hypothesis was rejected 
f. r the control offices, with the S.W. Broklyn district office's mean score 
significantly lower than the other two control offices. Although this difference 
was statistically significant, it amounted to only a mean difference of 1.06-1.43 
pointr on a 30-point scale. 

Since lower selection test scores were one possible explanation for this finding 
(see Table 7), an analysis-of-cov riance model was analyzed with selection test 
scores as the covarlate. This analysis indicated that when selection test scores 
were accounted for, the mean review test scores in the control district offices 
were no longer significantly different. Therefore, it was concluded that both 
training approaches were successful in delivering standardized training. 

Self-Report of Enumerator Performance 

Although asking for a self-report of performance typically produces unreliable 
data, such a question was asked on the post-training questionnaire to determine 
how such responses varied with the quantitative measures used in this study. 
As shown in Table 2, there were no significant differences in the response 
patterns of JPA or control enumerators. Tab' C in the appendix gives a similar 
breakdown for matched pairs of district offices. 



87 



TABLE 2 s. 
Enumerators' Self-Reported Job Performance 
Question: M How would ycu rate your own job performance so far?" 





N=-901 


N=549 




JPA 


Control 


Very Good 


37. 4* 


34.8% 


Good 


52.4 


54.5 


Fair 


9.4 


10.0 


Poor 


0.8 


0.7 



The breakdown 1n Table C for office pairs is of value because of the difference 
In response patterns found in the S. Oayton office. It is possible that factors 
unique to that office (e.g., field problems) affected performance. 

Self-Reported Average Dally Use of Manuals 

One expressed purpose of the JPA tralnlng'was to stress the use of the job aid 
manual during training both to simplify the presentation of procedural informa- 
tion and to provide the enumerators with the skills necessary to use the manual 
when 1t was required on the job. It was hypothesized that a simplified job 
manual would be used more frequently on the job, thereby resulting 1n closer 
adherence to procedures and higher quality data. Although no modifications were 
made to the Questionnaire Reference 8ook (QRB), an Identical question was asked 
about the QRB to determine how often this manual was also used. 

The means and standard deviations of manual (job aid or standard) and QRB use 
are reported in Table 3. It should be noted that this question was asked on 
the post-training questionnaire which was completed by an enumerator during 
first review, or approximately 2-4 days after nonresponse followup was started 
1n an enumeration district. Any differences 1n manual use attributable to 
training were expected onl> during this period because if ap enumerator's 
► ■Trformance was Judged deficient during first review, the crew leader was 
supposed to give on-the-job training which could have washed out any behavioral 
differences resulting from 'ifferent training approaches. 



TABLE 3 

Enumerator's Self-Reported Average Daily Use of the 
Enumerator Manual (Job Aid or Standard) and ORB 



Question: "On the average, how 


many 


times each 


day do you use-- 






JPA 


Control 




Mean 


SD 


Mean SD 


Manual (Standard or Job Aid) 


2.02 


2.13 


1.77 1.8. 


QRB 


1.04 


1.66 


1.27 1.59 


Total 


3.06 




3.04 
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It 1s interesting to note that both groups reportedly used their manual and QRB 
a total of about three times each day. To determine whether or not the means 
of reported use Offered, two different analyses of variance were computed, each 
making different assumptions about the data. 

The first nioael was an analys1s-of-variance model for nested factors (census 
district offices nested under training approach) described 1n Winer (1962, p. 184).!/ 
This model assumes no Interaction effect between training approach and district 
office. The ANOVA table is presented 1n Table D of the appendix and shows that 
both type of training and district office significantly affected manual use, *1- 
though the amount of explained variance was only four percent (square of multiple 
correlation coefficient, R*0.2). 

The second model analyzed was also an ANOVA model, but 1n this case it was 
assumed that a matched pair of district offices represented one office with 
two different kinds of training (I.e., a 3 x 2 model with district office and 
type of training as main effects). The results of this analysis are presented 
in Table E 1n the appendix. In this case an Interaction term was Included 1n 
the model and it was significant, Indicating that manual use varied depending 
on the district office pairing. Figure 1 in the appendix graphically represents 
this interaction and shows that the third pair - S. Dayton and W. Columuus - caused 
the Interaction. 

A comparison of Individual district office means for both manual and QR8 use 
is shown 1n Table 4. These analyses show that the greater overall manual 
use found for the JPA group was attributable to one JPA office, the S. Dayton 
office. 
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TABLE 4 

District Office Comparison of Means for Reported use of Manual and QRB 

(386) (117) (155) (214) (358) (201) 

*W. Queens S.W. Brook. *P1tts. W. Pitts E. *S. Dayton W. Col. 

Job Aid or 

Standard Manual 1.6(1.9) 1.7(1.8) 1.8(1.8) 1.8(1.9) 2.6(2.3) 1.8(1.7) 

W R 1.0(1.7) 1.4(1.7) 0.8(1.2) 1.3(1.7) 1.2(1.8) 1.2(1.5) 



* JPA offices 



NOTE: Means are based on number above district office name. Standard deviations are reported In 
parentheses next to mean. 

90 

91 
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Similar analyses were also performed for the reported use of the QRB. Table D.l 
in the appendix presents the ANOVA which uses a nested design and Table E.l the 
ANOVA which assumes equivalence of offices 1n a matched pair. In both these 
analyses the control district offices had a higher reported use of the QRB, al- 
though the design factors account for a trivial amount of the total variance 
(square of multiple correlation coefficient 1n both analyses is •01). Referring 
once again to Table 4, the control DO's had a higher reported rate of QRB use in 
two of the three office pairs. 

Several points are worth making about these findings. First, the reported daily 
use of job reference materials appeared quite low, especially considering that 
the enumerators were inexperienced. A concern must be expressed that manual 
design may be an Important factor only 1n training when enumerators are learning 
the job, since there may be a strong bias against using job reference materials once 
the actual enumeration begins. If this bias does exist, then when training 
1s designed, the assumption cannot be made that briefly covering a topic 1n 
training 1s adequate since the enumerator would have a manual to refer to on 
the job. Instead, a more reasonable assumption m1c,ht be that, 1f a skill or know- 
ledge 1s considered crucial to correct job performance, every effort must be made 
to ensure that it is mastered before the enumerator leaves training. 

Second, although the rate of QRB use In a JPA office was lower than Its control 
office In two of the three office pairs, the job aid manual contained some of the 
QRB Information that was expected to be used most frequently (i.e., the age con- 
version table and table of residency rules). Accordingly, JPA~trained enumerators 
in certain cases could have used the job aid manual instead of the QRB. 

Finally, the differences in reported rates of manual use, although statistically 
significant, would not appear to be of any practical significance. 

Cunulative Number of Qi estionnalres Completed by An Enumerator After Three Days 
of Work in Followup 1 

As part of progress reporting for follovwp 1, enumerators were supposed to con- 
tact their crew leader on a dally basis snd report the number of nonresponse 
questionnaires they had completed. This count w*s obviously a gross measure of 
production since the count was unverified and none of the questionnaires had gone 
'through an edit. 

Cumulative production after the first 3 days of work was of interest because 
crew leaders should h&ve conducted a first review after that period of time. 
Accordingly, this review was expected to eliminate any performance problems 
through either retraining or release of the enumerator. The mean cumulative 
production rates for the JPA and control offices are presented in Table 5. 
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TABLE 5 

Average Cumulative Production Figures 
After 3 Days of Work 

Mean Stan. Dev. 
JPA 24.9 16.4 

Control 27.2 18.3 

These figures are notable because in both groups production was far below pre- 
census expectations. Specifically, enumerators were told that they were expected 
/?J?!5r te 2 J"-J I 6 nn , n *PW* cases per 8-hour day, or 14 acceptable cases 
(includ ng short and long ,'onns)i/ per 8-hour day to earn targeted hourly wages. 
Obviously, neither group approached the expected average production of 48 cases, 
and it also seems likely that few piece-rate enumerators were earning the targeted 
wage, at least for the first few days of followup 1. 

Before analyses of the figures 1r, Table 5 are described, several possible co- 
variates that might have affected production will be discussed in turn. These 
factors Include different methods of paying enumerators, time of training, and 
selection test scores. 

Shortly after the start of followup I, field problems occurred in all participating 
district offices. Apparently, the most serious of these problems concerned errors 
In the Master Address Registers. For example, enumerators complained about 
duplicate listings, nonexistent housing, and delays 1n processi^late mail returns. 
These errors took time to find and correc* and some district M m\ switched 
their enumerators from piece-rate to hourly wages in order td^HHth resulting 
attrition and morale problems^!/ Unfortunately for the purposlHPthls study 
different pay sca.es could Introduce a serious confounding variable. Therefore, 
In an attempt to determine if method of payment differed between training groups 
data-capture clerks 1n each of the offices were instructed to record how an 
enumerator was paid, but the only Information that could be obtained was whether or 
not an enumerator worked on an hourly pay scale at any time during followup 1. 
■The percentage of enumerators w!,o were paid an hourly wage at any time during 
followup 1 1s reported 1n Table 6 for each district office. The data clearly 
show that district offices differed substantially in their reported use of hourly 
enumerators during followup 1. y 



1/ D-548, Followup 1 Enumerator's Manual (Decentralized), p. 112. 

1/ Even prior to these problems it was not planned for all enumerators to be 
paid piece-rate. Because of special circumstances 1n some enumeration 
districts, enumerators who worked in them were paid an hourly wage. 
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TABLE 6 



Percentage of Enumerators Who Were Paid 
Hourly at Any Time During Followup 1 



District Office 



Perce nt 



W. Queens 
Pittsburgh West 
S. Dayton 



0.0 
37.4 
28.7 



Average for JPA 



24.0 



S.W. Brooklyn 
Pittsburgh East 
W. Columbus 



0.4 
17.6 
49.0 



Average for Control 



22.0 



One possible reason for problems 1n the address registers cited by several crew 
leaders was that follovwp 1 started too soon 1n some offices; that 1s, 1t started 
before all late mall returns had been checked into the district office and their 
status changed from nonresponse to response 1n the address register.!/ Assuming, 
due to delays 1n questionnaire check-1n, the presence of a large number of Incor- 
rectly identified nonresponse addresses in an address register, two different 
effects on production can be hypothesized. First, since enumerators were told to 
complete a questionnaire - even though the respondent might claim to have mailad 
one 1n already - production might have been boosted since there would have been an 
Inordinate nuittoer of nonresponse addresses available. However, another possibility 
was that enumerators would grow tired of confronting respondents who already 
. claimed to have mailed in a questionnaire and, instead, would target those households 
for a return visit later 1n followup 1 when more mail returns had been processed. 
In this case, production would be lowered because visits to incorrectly identified 
nonresponse addresses would not contribute either to production or an enumerator's 
pay. In light of the production figures reported in Table 5, it seems that this 
latter hypothesis is better supported. 

As mentioned previously, two other variables that might also have affected produc- 
tion were time of training and selection test: scores. As noted in PERM No. 9, 
not all offices trained their followp 1 enumerators on the same dates (see Table 2 
in Appendix A in PERM No. 9). Therefore, it is likely that the more training was 
delayed, the less severe the questionnaire check-in problem would have been. 



2/ Because of the larger than expected mail response, questionnaire check-in in 
the district offices fell behind schedule. 
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Finally, although an attempt was made to match office pairs for this study, 
local differences could be expected to affect both the availability and 
quality of the work force that was recruited. Table 7 provides a breakdown 
which compares offices on the average selection test scores of their followup 1 
enumerators. 



TABLE 7 



Mean of Enumerator Selection Test Scores* by 
District Office 



District Office 



Mean 



Stan. Dev. 



W. Queens 
Pittsburgh West 
S. Dayton 



36.0 
37.7 
38.8 
TF3 



8.4 
8.4 
8.1 
O 



Average for JPA 



S.W. Brooklyn 
Pittsburgh East 
W. Columbus 



35.1 
38.7 
38.6 
T7TT 



8.8 
8.4 
8.4 
877 



Average for Control 



* Maximum score is 54. 
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To determine 1f the mean difference in Table 5 (which favored the control group) 
was statistically significant, average production scores for the JPA and control 
groups were compared using regression analysis to analyze the nested ANOVA design 
described earlier, except that training date and selection test scores were 
separately introduced as covariates.l/ Tables F.l and F.2 in the appendix present 
the results of these analyses. In all cases, the covariates were entered first 
in the regression model, followed by dummy variables for training approach and 
district office (district office was nested under type of training). 

Across all the models analyzed, method of training had the largest effect account- 
ing for approximately three percent of the total variance after the covariate 
had been partialed out. As expected, selection test scores had a positive, but 
small, effect on production. Table G in the appendix presents a breakdown of 
these production figures by district office. Except for the S. Dayton - W. Columbus 
pairing, control office production in each case exceeded that in the matched JPA 
office. 

Quality Control Enumerator Progress Report (Form D-170) 

Instead of the office edit o^ nonresponse questionnaires that occurred in centralized 
district offices, nonrespon * questionnaires in decentralized offices were reviewed 
by Quality Control enumerators. This edit by QC enumerators was essentially a 
check for last resortZ/ information, plus some additional information required on 
long-form questionnaires. Therefore, the QC enumerator's report included an assess- 
ment of a minimal level of quality for enumerator-filled questionnaires. Each 
QC enumerator was supposed to review on alternate days, the work produced by two 
different crews of enumerators. 



- 1/ Introducing more than one covariate resulted 1r a trivial Increase in the 
multiple correlation coefficient. Method of enumerator pay could not be used 
as a covariate because the exact time an enumerator was converted from piece- 
rate to hourly was not known in all cases. 

Z/ Last Resort information referred to a minimum number of population and housing 
items that needed to be filled for a questionnaire to be considered "acceptable. 
For example, on the short form it was the name of each persoi, plus at least 3 
of the following 4 population questions: relationship, sex, race, and marital 
status. The required housing items were: nurriber of living quarters, method of 
entering living quarters, plumbing facilities, and value of and commercial or 
medical use of one-family houses. 
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.or the purposes of this study, the first two reports filed for a crew leader 
during follovwp 1 were used for analysis. Table 8 presents the mean production 
figures for the JPA and control groups for the first two QC reports. Breakdowns 
of these tables by district office are provided in Tables G.2 and G.3 of the 
appendix. 

TABLE 8 

Average Number of Acceptable Questionnaires Reported 
On the QC Enumerator Reoorts 

Time 1 Time 2 

Mean Stan. Dev. Mean Stan. Dev. 

JPA 17.0 ,15.6 19.2 16.7 

Control 22.2 17.5 23.6 18.5 

To determine if these mean differences were significantly different on the two 
occasions, t.he analysis described in the previous section was repeated. Tables 
H.1 arid H.2 1n the appendix present the results of these analyses for the first 
qc enumerator report; Tables 1.1 and 1.2 present the results for the second QC 
enumerator report. 

Consistent with the analysis of the cumulative production data (Tables F.l and F.2), 
on both QC enumerator reports JPA-tralned enumerators produced fewer "acceptable- 
questionnaires than their counterparts In the control offices. Also consistent 
with the results for the cumulative production data, higher selection test scores 
tended to result In greater numbers of "acceptable" questionnaires on both reports. 
However, contrary to the analysis of the cumulative production data, individual 
district offices contributed significantly to a reduction in residual variance 
In both analyses. Finally, the later an enumerator was trained had no significant 
effect for the first QC report, but consistent with the cumulative production 
data, later training dates tended to result in fewer numbers of "acceptable" 
questionnaires produced on the second QC report. 

Although there is definitely a consistent pattern to the results obtained for 
the quantitative production measures used In this study, several facts affect 
the general izabllity of the results. First, together the explanatory design 
variables and covariates explain relatively little (4-12 percent) of the total 
variation present in the dependent variables, which were basically gross measures 
of production. Moreover, the large sample sizes of enumerators used contributed to 
finding statistically significant differences, but determining practical signifi- 
cance Is a more complicated issue that must consider a variety of other factors (e.g. 
additional followup required, quality of the data, coverage, and costs of enumerator 
attrition). 
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Second, almost all the production measures used had high rates of missing data, 
particularly in the control offices, which could have introduced a systematic 
Mas into the results. One possible explanation for these response differences 
between the JPA and control offices was that Field Operations Assistants (FOA's) 
in the JPA offices were aware of differences in the training materials being 
evaluated and expended more effort than their counterparts in the control offices 
to assist the evaluation by ensuring, for example, that crew leaders conducted 
first reviews and turned in the results. Failure to conduct an equivalent propor- 
tion of first reviews in the control offices could have resulted 1n artificially 
Inflated production scores since enumerator errors would have gone undetected 
and questionnaires accepted that otherwise might have required additional followup. 
The review by the QC enumerato r woul d not have rectified this situation since 
this check was basically only for Last Resort information. 

Third, considering the small number of district offices sampled, field problems 
(e.g., problems with the address registers) might have been worse in one group 
(JPA or Control) as opposed to the other. Problems with address registers, for 
example, definitely lowered Initial production scores because enumerators were 
often required to follow up addresses that either were duplicates or had already 
mailed in a questionnaire. Unfortunately, no data were collected on the differen- 
tial severity of field problems in the JPA and control offices, so no comparisons 
are possible. 



Record of First Review fD-185 ) 

The Record of First Review was a crew leader's first quality check of an enumer- 
ator s work and involved an examination of entries in the Master, Address Register 
(Part A) and an edit of five randomly selected questionnaires (3 short forms and 
2 long forms, see last page of appendix for an example of a 0-185) to check for 
improperly answered or skipped questions. 

Part A of the Record of First Review provided no specific instructions for evalu- 
ating an enumerator's work. If a crew leader thought tha,. performance was 
acceptable, the "Yes" column was marked for a given job task. If performance 
was not considered acceptable, the crew leader was instructed to retrain the 
enumerator and circle the 'X" in the "No" column to Indicate that the enumerator 
understood the correct procedure. Overall results for Part A were recorded in 
the Results box on the f.ont page of the 0-185, Record of First Review. 

Part B of the Record of First Review was essentially a tally sheet for errors 
found on the short or long forms. If more than 4 errors were found for a sample 
of five questionnaires, the crew leader was supposed to schedule another First 
Review. If there were more than 20 errors, the crew leader was supposed to 
instruct the enumerator to stop working until the situation was discussed with 
the FOA and a decision made to retain or release the enumerator. 

An analysis of the number of "No" boxes checked on the front of the 0-185 
Indicated little difference between the JPA and control groups. On an average, 
0.3 boxes (out of a possible total of 6) were checked per enumerator in both 
groups. Table 9 presents the results of a crew leader's overall evaluation of 
an enumerator's work based on a review of the items on the front page of the 
0-185. Evaluation results for enumerators did not differ significantly between 
the JPA and control groups. Slightly over 5S of the enumerators in the JPA 
group and 1% of the enumerators in the control group failed Part A of First Review. 
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TABLE 9 

Results of Crew Leader Evaluations Based on a General Review of the 
Master Address Register and Questionnaires 



(N-916) (N*936) 

JPA Cont rol 

Satisfactory 94.8% 92.7* 

Needs Improvement 4.6 5.8 

Unsatisfactory 0.7 1.5 



Table J in the appendix presents a breakdown of Table 9 by district office. 
This table 1s notable because, compared with other measures used in this 
study, there 1s little betweeo-of fice variation. 

The average number of short-form and long-form errors found during the crew 
leader's edit of 5 randomly selected questionnaires is reported in Table 10. 



TABLE 10 

Average Number of Edit Errors Reported by Crew Leaders 
During First Review 

JPA Stan. Dev. Control Stan. Dev. 

Short-Form Errors 1.1 2.0 1.1 1,8 

Long-Form Errors 1.1 2.1 1.0 1.6 



Although crew leaders were not required to do a detailed edit, the average 
number of errors reported seems so low that concern must be expressed about the 
validity of the figures. For example, based on edit results after the Oakland 
pretest,**/ average NA rates per questionnaire (items not completed that should 
have been) of 4.7 items for short forms and 44.6 items for long forms were 
reported. 

Table 11 presents the overall results for the crew leader edit during First Review. 
As noted in the table, slightly over 10% of the enumerators in each group failed 
Part B of First Review. Only 3.2% of the enumerators in both the JPA and control 
groups failed both Part A and Part B of First Review. 



!/ Figures supplied by Richard Griffin (SMD) were based on enumerator- filled, 
occupied housing units. 
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TABLE 11 



Results of Crew Leader Edit of a Sample of 
Short and Long Forms 



Satisfactory (0-4 e 
Needs Improvement (5-20 e 
Unsatisfactory (more than 20 e 





(N-922) 


(N=916) 




JPA 


Control 


rrors) 


89. 7* 


89. 8* 


rrors) 


9.5 


9.5 


rrors) 


0.8 


0.7 



Table K 1n the appendix presents a breakdown of Table 11 by district office. 
Unlike the results 1n Table J, there 1s more between-of fice variation with the 
Pittsburgh West office falling proportionally more enumerators than any of the 
other offices. It 1s possible that this aberration was caused by a unique field 
problem in that district office; specifically, approximately 17-20% of the origi- 
nal nonresponse cases 1n the office were actually late mall returns that the U.S. 
Postal Service failed to deliver until April 23, 1980 - a week after the start 
of followup 1. If enumerators attempted to obtain completed questionnaires from 
these addresses, 1t is likely that more resistance would have been encountered 
from respondents which could have resulted 1n a greater number of Incompletely 
filled questionnaires. Accordingly, proportionally more enumerators would have 
failed First Review. 



Record of Reintervlew, Form 0-158 

Completing the Record of Reintervlew required a crew leader to select a random 
sample of five completed nonresponse questionnaires for each enumerator and to 
telephone, 1f possible, the household to verify the household roster and address. 
However, this procedure, more than any other, was resisted by crew leaders and 
1n several offices was done for only about 20X of the enumerators (see Table A 
1n appendix). For example, three Field Operations Assistants (FOA's) provided 
personal letters explaining the lack of D-158's, Record of Reintervlew, from 
their office. Reasons given for not completing the form were that followup 1 
failed to start on schedule, or started 1n a piecemeal fashion with enumerators 
going Into the field at different times. Because of the delays and pressure to 
get work Into and out of the field, crew leaders reportedly did not have time to 
conduct the reintervlew. In fact, one F0A suggested that the QC enumerator should 
have had this responsibility. Moreover, when the reintervlew was attempted, the 
crew leaders reported resistance and even hostility from respondents who had 
reportedly mailed 1n their questionnaire, but were visited, sometimes more than 
once, by followup enumerators because of errors 1n the address registers. Crew 
leaders also reported that some respondents did not like to give information over 
the telephone and were suspicious of the crew leader's authority. Finally, other 
crew leaders disliked the reintervlew procedure because they felt that it put 
them in the position of having to question the Integrity of their enumerators. 
However valid those complaints might have been, the Record cf Reinterview did 
have the highest rate of missing data, ranging from 49.7% in W. Queens to 80.6% 
in W. Columbus. 
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Another problem with the Record of Reinterview was that there was reason to 
believe that procedures were not correctly followed even when ft was completed. 
For example, enumerators who failed the reinterview should have been rei nterviewed 
again with a new sample of questionnaires, but this apparently did not happen as 
often as requi red. 

On the average, 3 enumerators out of 100 failed reinterview in the JPA offices 
versus 5 out of 100 in the control group. Table L in the appendix reports the 
percentage of enumerators in each office classified by the nunfcer of questionnaires 
for each enumerator found to have household listing errors. Based on these 
results, it is estimated that the reinterview procedure identified 41 questionnaires 
across all six district offices that had household member listing errors. No 
attempt was made to estimate within-household coverage error. 

Other Variables of Interest 

The performance measures discussed so far were selected because of th^ r 
temporal proximity to training. Beyond a given point in the followup ^eriod, 
It was assumed that factors such as on-thfc-job training by supervisors, attrition 
or release of less productive or unmotivated workers, and other factors (e.g., 
quality of supervision) unique to district offices would obscure the effects of 
training, particularly because the performance measures used were only gross 
measures of production and quality. Still, they were the management tools used 
to judge the progress of the census. 

Definitely more removed from training, but probably indirectly affected by it, 
were other measures of a successful operation such as enumerator attrition and 
final closing dates for an office. Although no definitive conclusions can be 
made about these measures, patterns of results may emerge that either support 
or contradict earlier findings. For example, the percentage of enumerators who 
comoleted their assignments, the end of followup 1 in each office, the closing 
date of each office, and unemployment statistics for March - May 1980, are shown 
i n Table 12. 
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TABLE 12 

Percentage of Enumerators Completing Their Assignments, DO Closing 
Dates, and Unemployment Statistics 



a. 
b. 
c. 



JPA 

W. Queens 
Pittsburgh West 
S. Dayton 



Average 



Control 



a. S.W. Brooklyn 

b. Pittsburgh East 

c. W. Columbus 





End of 




Percent Unemployed 












Percent 


FU 1 


Closing Date 


March 


Ap ri 1 




41.1% 


July 3 


Oct. 16 


8.8 


7.5 


8.4 


74.0 


May 14 


Aug. 22 


7.0 


6.4 


6.5 


71.2 


May 23 


Aug. 26 


6.3 


6.8 


8.0 


62.7 












55.0 


June 21 


Sep. 26 


8.8 


7.5 


8.4 


58.9 


May 30 


Sep. 5 


7.0 


6.4 


6.5 


54.2 


May 20 


Sep. 5 


4.5 


4.9 


5.8 



Average 55.8 



Based on this table, there is obviously a relationship between the percentage of 
enumerators who completed their followup 1 assignments and the final closing date 
of a district office.!/ A significant (p<.05) Spearman rank correlation coefficient 
of 0.9 was computed (adjusted for tied ranks) for this comparison. However, it is 
Interesting to note that similar correlations computed for the 3-day production 
figures and the two QC enumerator reports were not statistically significant, 
(correlations ranged from 0.20 to -.55) indicating that initial production figures 
were not related to final office closing dates. Unemployment figures reported 
in Table 12 bear little relationship to enumerator attrition figures. For example, 
Pittsburgh West retained the highest proportion of its enumerators, but its 
unemployment rate dropped from March - May 1980, the period of time most li<ely 
to affect employment in followup 1 # 



This relationship was of primary interest because the percentage of enumerators 
who completed their assignments in followup 1 should be an indirect measure af the 
quality of work accomplished. Specifically, it can be hypothesized that less inter- 
viewer turnover results in higher quality work; moreover, less turnover means greater 
numbers of experienced workers would be available for followup 2. Accordingly, 
followup 2 should be shorter in offices with less turnover in followup 1, resulting 
in earlier closing dates. 
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Table 13 presents a breakdown of enumerator attrition for the JPA and control 
groups. Tiese data came from the final pay voucher filed for an enumerator 
during followup 1. Crew leaders were directed to enter a reason on the pay 
voucher explaining why the enumerato r was no longer worki ng (assignment completed, 
released, etc.)- If a reason was not on the form, data-capture clerks were 
instructed to contact either the crew leader or FOA to determine the reason. 
Table M in the appendix presents a breakdown of these reasons by district office. 



TABLE 13 



Analysis of Reasons for Enumerator Separations 





(JM140) 


(N»1160) 


Reason 


JPA 


Control 


Assignment completed 


62.7% 


55.8% 


Pay dissatisfaction 


3.2 


1.7 


Work dissatisfaction 


4.2 


10.9 


To take another job 


7.6 


6.2 


Poor performance 


4.9 


10.7 


Other 


17.4 


14.7 



Note: Reasons for attrition were available for 96.9% of the JPA enumerators and 
86.0% of the controls. 



The breakdown above should be interpreted cautiously because the obtained 
distribution may not reflect true causes of attrition. For example, as noted 
in a previous report (PERM No. 9), enumerator comments about pay led the list 
of improvements suggested for future censuses. Moreover, inadequate pay might 
have caused an enumerator to look for and find alternative employment or, if 
the enumerator quit because of a combi na^-'on of factors (one of which was pay), 
the action might have been interpreted as work dissatisfaction by a crew leader 
or FOA. 

The difference in attrition between the groups in Table 13 attributable to 
" poor performance is noteworthy, not because the difference favors the JPA group, 
but because the difference was caused by one control office - S.W. Brooklyn - 
that released approximately 20% of its enumerators because of poor performance. 
It might only be a coincidence, but this office also had the lowest mean selec- 
tion test scores of all participating offices (see Table 7). 

Method of Payment and Enumerator / t rition 

In the first memorandum describir,^ ne results of this study (PERM No. 9), it 
was noted that 3 of the 4 most frequently mentioned improvements to the enumer- 
ator's job dealt with pay. Accordingly, attrition was examined controlling for 
method of payment. To reiterate, if an enumerator was classified as an hourly 
worker, it meant only that at some point during followup 1 the enumerator was 
paid an hourly wage. It was possible, in certain cases, that hourly enumerators 
actually worked most of followup 1 as piece-rate workers. Unfortunately, data 
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on the conversion of piece-rate enumerators to hourly were collected by only 
one office.iii/ Table N in the appendix shows the cumulative percentage of 
enumerators who were reappointed to hourly wages fcr the S. Dayton district 
office. Followup 1 ended in late May in this office. In this office the 
greatest number of reappointments to an hourly wage occurred after approximately 
two weeks of followup 1. 

Considering this study's limitations in the definition of hourly workers, 
Figure A graphically compares the percentage of enumerators who completed their 
assignments and were paid hourly or piece-rate. Table 0 in the appendix presents 
the same data in tabular form. Two district offices were excluded because less 
than 1% of their followup 1 enumerators were paid hourly. 



FIGURE A 

Percentage of Enumerators Completing Their Assignments 
as a Functiort of Method of Payment 



Percentage of 
Enumerators 
Completing 
Assignment 




Hourly 



Piece-rate 



Pitts. E. *P1tts. W. W. Col. *$. Dayton 



*JPA office 



Table 14 presents a breakdown of the reasons for enumerator attrition controlling 
for method of payment. Ch1-square comparisons for both groups were significant; 
I.e., X* » 112.7, 5 df, p * 0.00 for the JPA group, and X? = 79.7, 5 df, p ■ 0.00 
for the control group. 



12/ These data were not required in original data-collection speclficatl 
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TABLE 

Reasons for Enumerator Separations Classified 

JPA 





(880) 

PR 


(260) 

H 


Assignment completed 


54. 5% 


90. 4% 


Pay dissatisfaction 


4.1 


0.0 


Work dissatisfaction 


5.3 


0.4 


To take another job 


9.5 


1.2 


Poor performance 


5.7 


2.3 


Other 


20.8 


5.8 


PR - Piece rate 


H - Hourly 





14 

By Training Approach And Method of Payment 



Cont rol 



(924) 


(236) 


PR 


H 


50.2X 


77.5% 


2.1 


0.4 


13.0 


2.5 


5.5 


8.9 


13.0 


1.7 


16.2 


8.9 
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In both Figure A and Table 14, proportionally more enumerators who were paid 
hourly completed their assignments. These results might also help explain a 
previous finding; specifically, the significant relationship between the percentage 
of enumerators who completed their assignments arri the final closing date for an 
office. It is possible that paying enumerators an hourly wage reduced attrition 
which helped maintain higher uniform productivity for an office which, in turn, 
led to earlier closings. 

Intercorrelations Among Performance Variables 

In any large-scale study, the reliability and validity of measures are an 
important concern. In this study, both were probably most affected by crew 
leader and QC enumerator adherence to procedures, as well as by systematic 
biases that may have existed in either the JPA or control group. Although 
reliability and validity were not directly assessed, they can be indirectly 
evaluated by examining performance variables and their intercorrelations 
(Pearson product moment). 

For example, it is reasonable to expect that production after three days and 
the number of questionnaires passing the first QC enumerator report, should be 
intercorrelated since both were obtained at about the same point in followup 1. 
Table 15 presents the intercorrelations of many of the variables discussed in 
this report. The correlations cannot be directly compared since .hey are based 
on different cases due to patterns of missing data. 



TABLE 15a. 

Intercorrelations Amo.ng Performance Variables, Selection Test 

Scores, and Final Review Test Scores for the JPA Group 

Selection Final Review 3-Day First Second 

Test Test Production D-170 0-170 

Selection Test 0.45 0.10 0.06* 0. 10 

Final Review Test 0.09 0.14 0.08 

3-Day Production 0.42 0 32 

First D-170 * 0 ; 35 



* Not significant, p<.05 



TABLE 15b. 

Intercorrelations Among Performance Variables, Selection Test Scores, and 
Final Review Test Scores for the Control Group 

Selection Final Review 3-Day First Second 

Test Test Production D-170 D-170 

0.43 0.1 1 0.09 0.08 

0.09 0.12 0.08 

0.51 0.40 
0.28 

100 



Selection Test 
Final Review Test 
3-Day Production 
First D-170 
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As expected, cumulative enumerator production after 3 days of work was positively 
and moderately correlated with the number of questionnaires that passed the first 
QC enumerator edit. The correlation was 0,42 in the JPA grouD and 0.51 in the 
control. Likewise, positive, but lower correlations were found between cumulative 
production after three days and the number of questionnaires that passed the second 
QC enumerator edit. Contrary to expectations, the first and second QC enumerator 
reports were not highly correlated. Correlations ranged from 0.28 in the control 
group to 0.35 in the JPA group. 

In both groups, selection test scores were moderately correlated with scores on 
the final review tests used in training, but correlations with the three production 
measures were weak and, in the JPA group, one (with the first QC enumerator report) 
was nonsignificant. Similarly, scores on the two final review exercises used fn 
training were only weakly correlated with the production measures. 

Although almost all the intercorrelations in Table 14 are significant, many 
are so low as to-be~o£ no practical significance. Especially bothersome are 
the weak relationships found between the aptitude and knowledge measures - i.e., 
the selection test scores and review test scores - and the three production 
measures. Moreover, the three production measures are, at best, only moderately 
intercor related which raises concern about their validity. Several possible 
interpretations exist to explain these results. 

First, it could be argued that the content of the selection test and final review 
test lacked relevance to the enumerator's job and factors important for success. 
However, a second interpretation is that the performance measures were, at best, 
only imperfect measures of performance that were insensitive to differences in 
the quality of an enumerator's work and bore little relationship to the extent 
of followup work required before a district office cculd be closed. The non- 
significant correlations reported earlier between the three production measures 
and district office closing dates tend to support this argument. 

A third possibility, at least for correlations involving the review test, is that 
correlations were lowered by a restriction of range on the test scores. For example, 
an average enumerator got 87% of the test items correct in the JPA group and 90% 
correct 1n the control group. 

Finally, 1t is likely that much of the variability in the three production 
measures wa* either of a random nature or due to factor? unrelated to training 
quality or the individual attributes of an enumerator. For example, assuming a 
large number of duplicate addresses or unprocessed late mail returns in a Master 
Address Register - common occurences in many offices at the start of followup 1 - 
by chance alone an enumerator could have visited an Inordinately large number 
of problem addresses during either of the sequential 2-day periods that served 
as input for the QC enumerator reports. Accordingly, large performance differ- 
ences could have resulted between the first and second QC enumerator reports. 

Unfortunately, for the purposes of this study, the period of time of most 
interest in followup 1 for assessing the effects of different training approaches 
was also the period most plagued by field problems that confounded comparisons 
between the JPA and control groups and, therefore, preclude making any definitive 
statements about the effects of alternative training on enumerator performance. 
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Recommendations 



As noted 1n the first memorandum describing the results of this study (PERM No. 9), 
the primary purpose of the Alternative Training Experiment was to develop a 
method of training that Incorporated the advantages of verbatim training, but 
that eliminated most of the disadvantages. Accordingly, an attemot was made - 
primarily through the use of job aids - to simplify the presentation of procedural 
Information in both the enumerator field manual and accompanying training. 
However, contrary to standard census reference material , the job aid manual used 
In this study was designed to be an Integral part of training, so much so, 1n 
fact, that 1t would have been difficult to use 1f an enumerator had not gone 
through formal training. In addition, training activities were designed con- " 
currently with the job aid manual to reduce the amount of lecture and Increase 
trainee participation, to introduce a variety of learning activities (e.g., work- 
book exercises, practice Interviewing, audiovisuals, problem solving) for combating 
monotony and boredom, and to use performance checks (e.g., quizzes, H*scuss1on 
periods, exercises, formal review test) that enabled the trainer to i „n1tor the 
learning process and ensure that all trainees left training with at least a 
minimum set of knowledge and skills. 

The success of the alternative training in achieving some of these objectives 
was clear-cut. The standard enumerator field procedures manual was shortened 
from 129 to 62 pages and used for Instructional purposes in training that were 
not practical with the standard manual. Moreover, enumerator evaluations of 
the training favored the alternative training on measures such as quality, 
comprehenslbllity of training materials, and length of training. This study, 
therefore, demonstrated that census manuals and accompanying training could be 
significantly simplified without major Increases 1n training costs and using 
existing Bureau facilities. However, the attainment of other objectives was less 
clear. For example, this study did not demonstrate that better enumerator reac- 
tions to training translated to higher quality performance or production. In 
fact, production measures, acknowledging problems with them, tended to favor 
the standard census training. 

Although no definitive statements about the effects of training on performance 
can result from this study, several suggestions for training development and 
research are presented based on experiences garnered during the course of this 
experiment. They are: 



1. More preparation time 1s required for experiments of this nature. 
Further, small-scale experimentation with alternative training 
methods should proceed prior to large-scale field experimentation. 

a. In line with this recommendation, efforts should be continued 
to investigate alternatives or supplements to verbatim training. 
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b. Also, experimental comparisons of training alternatives could 
take place during census pretests. Dress rehearsals could 
further be used to refine existing training packages and to 
deal with unforeseen problems. 

2. All decennial training packages should be designed with at least two 
hours of flexible time.il/ This time could then be used to deal with 
both unforeseen field problems and situations unique to certain district 
offices (e.g., large numbers of seasonal units, large apartment complexes 
with poorly marked units, etc.). It was disconcerting to read in a 1970 
evaluation reporti£/ that problems most frequently cited by enumerators 
were mistakes in the address registers, uncooperative people, and 
respondents saying that a questionnaire had been mailed 1n. These same 
problems also occurred frequently 1n the 1980 Census which leads to the 
next recommendation. 

3. Training packages should present a more realistic picture of the work 
s1tuat1on.il/ Both training approaches in this study presented an 
Idealistic picture of an enumerator's work. Accordingly, both failed 
to prepare enumerators for the prevalent problems resulting from Incom- 
plete and m1xed-up address registers. Carefully structured training 
which assumes correctly prepared materials may be counterproductive 

and lead to worker cynicism and distrust about the integrity and competence 
of training when the work situation 1s not as described or taught. 

4. The design of training and field procedures manuals should be an 
Integrated effort. Ideally, the same person would write both the 
manual and the training guide, but these functions could be separated 
if a formal means of communication was established and all manuals 
(Including the QRB and administrative manuals) could be revised after 
training "dry runs" or tests to see if people learned the skills being 
taught. 

5. A structured approach (i.e., the JPA model) should be used to prepare 
all training materials. 

6. Persons responsible for developing procedures should attempt to present 
them in a job aid format. Or, procedures could be given to technical 
specialists who would rewrite them in a job aid format. This recommen- 
dation necessarily applies only to procedures suitable for such a format. 



Uj These recommendations were originally suggested by Donny Rothwell. 

11/ Inderfurth, Gail. Results from the 1970 Enumerator Variance Study 
Post-Enumeration Questionnaire. El 8 No. 40 
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7. All procedures should receive input from training specialists as to 
their trainability given time, monetary, and other constraints. Addi- 
tional input from field specialists would also be required to deter- 
mine if procedures, as written, could be successfully implemented in 
the field, 

8. To the maximum extent possible, manuals used in decennial training 
should be consolidated, integrated into the training, or both. 

For example, there was no apparent reason why an administrative manual 
was required for decennial followup 1 enumerators. Such a manual wa 
totally excluded from the alternative training, yet JPA enumerators 
reported fewer problems filling payroll (See PERM No, 9) forms than 
control enumerators. 

Finally a point that should be reiterated is that it was not the job aids alone 
which resulted in the differences in enumerator reactions found in this study. 
Of more importance was the process used to produce the job aids which avoided 
many of the bureaucratic roadblocks to communication present in the production 
of standard census traininq. It is likely that a different person, or group 
of persons, would have produced somewhat different job aids, but as long as the 
accompanying training was developed concurrently, effective training should have 
resulted. 

Bureaucratic divisions of labor created communication problems because in the 
planning and conduct of a census procedures changed frequently, were sometimes 
left vague, or were poorly understood by the writers responsible for designing 
training. Moreover, training specialists often observed awkward, diff icult-to- 
teach, or nonproductive procedures which were difficult to modify because the 
originator of the idea was several levels removed in the bureaucracy. 
Job-performance-aided training deals effectively only with the communication 
problems that occur in the preparation of manuals and training. As noted in 
recommendation 7, other channels need to be established so that feedback from 
writers about the trainability of procedures will also be considered. 
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TABLE A 



Percentage of Enumerators In Each District Office with 
Usable Performance Data 





District 
Office 


No. 
Completing 
Trai n1ng 


Sel f-Report 
E Perf 


Sel f-Report 
Manual Use 


Sel f-Report 
QRB Use 


Rev. Test 
Score 


Prod 
after 
3 Days 


0-170(1) 


D-170{2 


a. 


W. Queensi/ 


372 


♦100.0X 


♦100.0X 


♦100.0X 


78.2% 


94. 6% 


89. OX 


80. IX 


b. 


Pittsburgh W. 


324 


48.8 


47.8 


47.8 


80.6 


92.9 


91.7 


77.8 


c. 


S. Dayton 


459 


80.2 


78.0 


78.0 


98.7 


90.6 


80.4 


79.5 


a. 


S.W. Brooklyn*/ 


449 


26.7 


26.1 


26.1 


99.1 


92.7 


96.4 


72.6 


b. 


Pittsburgh E. 


310 


69.4 


69.0 


69.0 


87.4 


71.9 


77.1 


58.4 


c. 


W. Columbus 


516 


41.5 


39.0 


39.0 


68.0 


98.3 


39.7 


36.4 



17 JPA district offices 
1/ Control district offices 
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These figures were Inflated by clerical error because clerks failed 
to record the names of all enumerators who had completed training 
which provided the base reported in column 2 (No. completing training). 
Since the self-report measures came from the anonymous post-training 
questionnaire, 1t was not possible to determine which enumerators 
actually completed the measures. For the other evaluation measures 
shown In this table, it was possible to link a name with a performance 
figure. 
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TABLE A (Cont'd.) 

Percentage of Enumerators in Each District Office with 
Usable Performance Data 



District 
Office 



a. W. Queensl/ 

b. Pittsburgh W. 

c. S. Dayton 

a. S.W. Brooklyn?/ 

b. Pittsburgh E. 

c. W. Columbus 



First 
Review 



82. 8X 

68.8 

83.9 

73.3 
62.3 
80.2 



Relntervlew 

50. 3% 

22.5 

36.6 

52.1 
20.0 
19.4 



i/ JPA district offices 
2/ Control district offices 



a - Pair 1 
b - Pair 2 
c - Pair 3 



9 
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TABLE B 



a. Analysis of Variance for JPA Final Review Tests 



Source df Sum of Squares Mean Square F jo 

Between 2 26.7 13.36 .57 .57 

Within 249 5822.5 23.38 

TOTAL 251 5849.2 



| b. Analysis of Variance for Control Final Review Tests 

l 

4 

I 

! Source df Sum of Squares Mean Square £ £ 

i Between 2 94 . 4 46. 19 8. 51 .0003 1/ 

Within 249 1352.1 5.43 

i TOTAL 251 1444.5 



1/ Using the Scheffe post-test procedure, average scores in the 

S.W. Brooklyn district office were found to be significantly lower 
(p< .05) than the other two. 
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TABLE C 

District Office Comparisons for Self -Reported Job Performance 

question: "How would you rate your own job performance so far?" 

(N-375) (N=120) (N=158) (N=215) (N=368) (N=214) 

*W. gueens S.W. Brooklyn *P1ttsburgh W. Pittsburgh E. *S. Dayto n W. Columbus 

Very Good 41. IX 32. 5X 50. OX 31. 2X 28.3X 39. 7X 

Good 49.1 51.7 44.9 56.7 59.0 c 3.7 

Fair 8.5 15.0 5.1 10J 12.2 6.5 

Poor 1.3 0.8 0.0 1.4 0.5 0.0 



* JPA district office 
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TABLE D 

Analys1s-of-Variance Table (Nested Design) for Ma-\ial Use 



Source df 

Tra1 n1ng Approach 1 

District Offices 4 

Residual 1431 



Sum of Squares 
20.2 
200.1 
5612.7 



Mean Square 
20.2 
50.9 
3.9 



F 

5.13* 
12.70* 



* p< .05 
R 2 * 0.04 



TABLE D.l 

Ana1ysis-of-Var1ance Table (Nested Design) for QRB Use 



Sou rce 



Training ApD roach 
District Offices 
Residual 



df 

1 

4 

1431 



Sum of Squares 
19.1 
27.5 
3795.3 



Mean Square 
19.1 
6.9 
2.7 



F 

7.17* 
2.58 



* p< .05 
R 2 » 0.01 
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TABLE E 

Analysls-of-Varlance for Manual Use Assuming Equivalence of JPA and Control Offices In Matched Pairs 



Source 



Explained 
Residual 



df 



Main Effects 3 

District Office 2 

Training Approach 1 

Interaction 2 



5 

1425 



Sum of Squares 

167.2 
146.9 
25.7 

52.3 

219.5 
5613.5 



Mean Square 

55.7 
73.5 
25.7 

26.1 

43.9 
3.9 



14.1 
18.7 
6.5 

6.6 

11.1 



Significance 

.000 
.000 
.011 

.001 

.000 
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TABLE E.l 

Analysi?-of-Var1ance for QRB Use Assuming Equivalence of JPA and Control Offices 1n Matched P*1rs 

Source df Sum o\ Squares Mean Square F Significance 

Main Effects 3 28.0 9.3 3.5 .015 

District Office 2 8.9 4.4 1.7 .189 

Training Approach 1 20.4 20.4 7.7 .006 

Interaction 2 18.6 9.3 3.5 .031 

Explained 5 46.6 9.3 3.5 .004 

Residual 1425 3795.3 2.7 

R 2 « 0.01 
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Figure 1. 



Mean Reported Manual Use Assuming 
Equivalence of JPA and Control Offices In Matched Pal 




-♦ 1 v 

1 2 3 

District Office 



District Office 

1 - W. Queens and S.W. Brooklyn 

2 - Pittsburgh W. and Pittsburgh E. \0 

3 - S. Dayton and W. Columbus 



TABLE F.l 

Analysis-of-Covariance Table for Production 
(Selection Test Score as Covariate) 

R2 



Source 


df 


SS 


MS 


F 


Beta 


Chanqe 


Selection Test 


1 


1 ,933.9 


1,933.9 


7.4* 


.073 


.005 


Training Approach 


1 


11,407.7 


11,407.7 


43.4* 


-.055 


.032 


District Office 


4 


2,683.6 


670.9 


2.6 




.008 


Residual 


1,294 


340,121.5 


262.8 








R2 » 


.05 , 


* p< .05 











TABLE F.2 

Analysis-of-Covar1ance Table for Production 
(Training Date as Covariate) 

R2 



Source 


df 


SS 


MS 


F 


Beta 


Change 


Training Date 


1 


1,831.7 


1,831.7 


6.9* 


-.050 


.005 


Training Approach 


1 


11,348.4 


11,348.4 


43.0* 


-.095 


.032 


District Office 


4 


1,639.8 


410.0 


1.6 




.005 


Residual 


1,294 


341,326.7 


263.8 









R 2 * ;04 , * p< .05 
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TABLE 6.1 



a. 



District Office Comparisons of Average Production 
After 3 Working Days 



District Office Mean Stan Dev 

W. Queens 22.0 17.5 



b. Pittsburgh West 25.3 16.1 

c. S. Dayton 27.1 15.3 

a. S.W. Brooklyn 28.3 19.1 

b. Pittsburgh East 31.6 17.1 

c. W. Columbus 24.4 i7.6 



TABLE G.2 

District Office Comparisons of Performance On First QC Enumerator Report 

District Office Mean Stan Dev 

a. W. Queens 20.3 15.8 

b. Pittsburgh West 20.2 15.4 

c. S. Dayton H.6 13.9 

a. S.W. Brooklyn 22.3 17.5 

b. Pittsburgh East 27.5 19.3 

c. W. Coluri-: 15.9 12.9 



TABLE G.3 

District Office Comparisons of Performance On Second QC Enumerator Report 

District Office Mean Stan Dev 

a. W. Queens 23.9 16.4 

b. Pittsburgh West 19.7 15.9 

c. S. Dayton 15.0 16.4 

a. S.W. Brooklyn 23.1 17.8 

b. Pittsburgh East 30.4 21 > 

c. W. Columbus 17.9 13.3 
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TABLE H.l 

Analysis-of-Covariance Table for the First QC 
Enumerator Report in Followup 1 
(Selection Test Score as Covariate) 

R 2 



Sou rce 


df 


SS 


MS 


F 


Beta 


Change 


Selection Test Score 


1 


2,429.6 


2,429.6 


10.7* 


.099 


.007 


Training Approach 


1 


11,922.7 


11,922.7 


52.6* 


-.110 


.035 


District Office 


4 


29,529.9 


7,382.5 


32.6* 




.088 


Residual 


1,294 


293,104.6 


226.5 








R2 » 


0.13 , 


* p< .05 
TABLE 


H.2 








Analysis-of-Covariance Table for the First 
Enumerator Report in Followup 1 
(Training Date as Covariate) 




R2 
Change 


Sou rce 


df 


SS 


MS 


F 


Beta 


Training Date 


1 


619.3 


619.3 


2.7 


.033 


.0C2 


Training Approach 


1 


11,691.7 


11,691.7 


51.1* 


-.074 


.035 


District Office 


4 


28,594.3 


7,148.6 


31.2* 




.085 


Residual 


1,294 


25.5,081.5 


228.8 








R2 a 


0.12 


* p< .05 
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TABLE 1.1 

Analysis-of-Covariance Table for the Second QC 
Enumerator Report 1n Followup 1 
(Selection Test Score as Covariate) 

R2 





Sou rce 


df 


SS 


MS 


F 


Beta 


Change 




Selection Test Score 


1 


2,553.5 


2,553.5 


9.5* 


.090 


.007 




i ra i n i ng . »pproacn 


i 


7 CTC 7 
/ ,0/0. / 


7 C 7C 7 
/ ,0/0. / 


28.5* 


-.060 


• 020 




District Office 


4 


co, 507.7 


6,626.9 


24.6* 




.069 




Residual 


1,294 


348,621.5 


269.4 








• 


R2 » 


.10 


p< .05 
. TABLE 


1.2 






- 




Analysis-of-Covariance Table for the Second 
Enumerator Report in Followup 1 
(Training Date as Covariate) 


QC 


R2 
Change 




Sou rce 


df 


SS 


MS 


F 


Beta 




Training Date 


1 


2,709.5 


2,709.5 


10.0* 


-.039 


.007 




Training Approach 


1 


7,638.3 


7,638.3 


28.1* 


-.089 


.020 




District Office 


4 


23,795.8 


5,949.0 


21.9* 




.062 




Residual 


1,294 


351,215.8 


271.4 










R2 » 


.09 , 


p< .05 
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TABLE J 



District Office Comparisons for Part A Results on the 
Record of First Review 



(N-308) (N-329) 
*W. Queens S.W. Brooklyn 



(N=223) (N-193) 
*P1ttsburgh W. Pittsburgh E. 



(N=385) (N=414) 
*S. Dayton H. Columbus 



Satisfactory 92. 2% 

Needs Improvement 6.8 
Unsatisfactory 1.0 



90. 31 
6.7 
3.0 



93. 7% 
6.3 
0.0 



93. 8% 
4.7 
1.6 



97.4% 
1.8 
0.8 



94. 2X 
5.6 
0.2 



* JPA office 
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TABLE K 



District Office Comparisons for Part B Results on the 
Record of First Review 



(N-303) (N=329) 
*W. Queens S.W. Brookly n 



Satisfactory (0-4 errors) 90.1% 

Needs Improvement (5-20 errors) 9.2 

Unsatisfactory (More than 0.7 
20 errors) 



89. IX 
9.4 
1.5 



(N=234) 
*P1ttsburgh W. 

82. 5X 
17.1 
0.4 



(N«192) 
Pittsburgh E . 

89. IX 
10.9 
0.0 



(N=385) (N=395) 
*S. Dayton W. Columbus 



93. 8X 
5.2 
1.0 



90. 9X 
8.9 
0.3 



* JPA office 
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TABLE L 



Percentage of Enumerators With Different Numbers of 
Questionnaires That Failed Reinterview 



(N-187) (N=234) 
*W. Queens S.W. Brooklyn 



(N=73) (N=62) 
♦Pittsburgh W. Pittsburgh E. 



(N=168) (N=100) 
*S. Dayton W. Columbus 



Number of 


0 


96. 3X 


95. 3% 


97.3X 


96. 8X 


97. OX 


95. OX 


questionnaires 


1 


3.2 


3.4 


2.7 


3.2 


1.8 


3.0 


failing 


2 


0.5 


0.9 


0.0 


0.0 


0.6 


2.0 


reinterview 


3 


0.0 


0.4 


0.0 


0.0 


0.0 


0.0 




4 


0.0 


0.0 


0.0 


0.0 


0.6 


0.0 






MPA office 
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10) 
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TABLE M 

District Office Comparisons of Reasons for Enumerator Separations 



(N-350) (N*500) 
*W. Queens S.W. Brooklyn 



(N=315) (N-302) 
♦Pittsburgh W. Pittsburgh E. 



(N=475) (N=358) 
*S. Dayton W. Columbus 



Assigning completed 
Pay dissatisfaction 
Work dissatisfaction 
To take another job 
Poor performance 
Other 



41. it 
9.4 
8.6 

16.3 
5.1 

19.4 



55. OX 
0.8 

15.4 
2.2 

20.6 
6.0 



74. OX 
0.0 
1.6 
4.8 
o.O 

13.7 



58. 9X 
3.3 
; 0 
4.0 
3.3 

25.5 



71. 2X 
0.6 
2.7 
3.2 
4.0 

!3.1 



54. 2X 
1.7 
9.5 

13.7 
3.1 

17.9 



* JPA office 
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TABLE N 

Enumerator Reappointments to Hourly Pay in the S. Dayton District Office 

(N-477) 



Percentage Cumulative 
% % 



April 14 9.2 9.2 

April 21 0.8 10.0 

April 28 11.5 21.5 

May 5 5.5 27.0 

May 12 1.5 28.5 

May 19 0.0 28.5 



Note : Followup 1 ended May 23 in this office. 
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TABLE 0 



Percentages of Enumerators Completing The1 r Assignments Classified 
Training Approach and Method of Payment 





PR 


H 


S.W. Brook 
*W. Queens 


55.0(498) 
41.1(350) 


50.0(2) 
0.0(0) 


Pitts E. 
*Pitts W. 


54.3(245)' 
64.6(192) 

• 


78.9(57) 
88.6(123) 


W. Col. 
*S. i/ayton 


31.5(181) 
62.7(338) 


77.4(177) 
92.0(137) 


PP - PiecK Rate , 


, H - Hourly 





Note: Bass of percentages is in parentheses. 
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»o«m 0-U5 



u.s. ae^A^TMCNT or COMMERC* 

• UHtAU Or TMt C*NSU4 



RECORD OF FIRST REVIEW 

20th Decennial Census - 1980 



1. District Office name J Code 



2. Crew leader name 



4. Enumerator name 

~7&cry J^y 



3. CLD No. 



5. EO No. 

J&7 



n General Instructions 

Complete a first review for each enumerator. A first review consists of two parts: Part A, 
Master Address Register and Questionnaire Review and Part B, Field Quality Check (on 
reverse side of this form). Instructions for these reviews are in your Crew Leader Manual. 
After you complete the review, take the action indicated at the end of Parts A and B. 



Port A - Motter Address Register ond Questionnaire Review 



Review the Master A4dre%s Ref /ster and Questionnaires and rate the enumerator on the 
following aspects of the job. 



Is the enumerator 


Mbrk (X) one 


Explanation for 1 
"No" entries | 




No 


1. Using the correct form type? 


y< 






2* Filling out the address label correctly 
so that the entries in the address label 
agree with the entries on the Address 
Listing Pages? 




s> 




3* Entering his/her name and date and the 
respondent's telephone number on the 
back of each questionnaire? 


y 






4, Making neat ant legible entries on 
the questionna.re? 








5e Completing columns (10) and (13) for 
each nonresponse case enumerated? 


X 






6* Entering daily progress figures on the 
cover of the Master A4drtss Register? 


X 







7. Describe any differences from prescribed procedures. Also enter any general remarks you have 
concerning the enumerator's work. 



Oiscuss incorrect procedures with the enumerator, 
the number of actual errors* 



Base the evaluation below on the discussions and 




□ SATISFACTORY j Is necessary 



r 

J The enumerator should continue working and nc additional review 



It 



jSfcNEEDS 

IMPROVEMENT 



The enumerator should continue working, but an additional review must 
i be made. Arrange to meet the enumerator and review the work for all 
[aspects marked l- No M in First Review. 



i The enumerator is unable to do the job as prescribed. Tell the 
{enumerator to STOP working until you are able to see him/her 



m . }Q UNSATISFACTORY tegain. Oiscuss the situation immediately with your supervisor. 



no 



Part B - Quality Check 



Using the instructions in your Crew Leader Manual, select a samoU of 3 Questionnaires so tnat 
a)! are for occupied households and at least 2 are Ion* forms. Review the sampie question- 
naires as prescribed below, note any frors in column (b), ana then take the action indicated in 
the "Results' 1 section. 



Review of sampje questionnaires 
U) 


Tally of errors 
<b> 


SHORT 
FORMS 


Make one ta.iy each time a question or item in tne fcl low- 
ing groups is blank or incomplete and requires an answer: 

• population questions on pages 2 and 3 

• housing questions on page 3 

• items in FOR CENSUS USE ONLY 
box on page 3 




LONG 
FORMS 


Make one tally each time a question or item in the follow- 
inggroups is blank or incomplete and requires an answer: 

• population questions on pages 2 and 3 

• housing questions on page 3 

• items in FOR CENSUS USE ONLY 
box on page 3 

• sampie housing questions on pages 4 and 5 

• *sample population questions on pages 6-! 9 

*NOT : If no sample population pages are completed 
for a person, make 5 tallies, and go on to 
the next page. 


/ 
/ 

>//// //// 


TOTAL ERRORS ► 




Notify your Field Operations Assistant if the number of errors exceeds 20. Otherwise, retrain the enumerator 
on the questions or items where errors made. Se sure mat the enumerator corrects any errors on the sample 
questionnaires, enter "edited by (your initials)'* be!ow the address laoel of the 5 questionnaires edited. 


i ; 

1 [n Q-4 trrors < w«ti factory. No further, action required. 


aettn ' ^ r ' N##<i * improvement. Schedule another review and complete a 
RESULTS errors -j new Form D-185. 


i « Unsatisfactory, The enumerator is unable to do the job as prescribed. 
| fj More then ! Tell tne enumerator to STOP working until you are able to se- him/her 
! 20 trrors \ again. Discuss the situation immediately witn your supervisor. 



FOAM 0»M§ lf-l»-?*l 



